Approach in 358c834 didn't worked out because annotations are assigned to the drawing after assign_class and we can't check in assign_class whether it's going to be part of the drawing.
Apparently we can't trust `obj.data == None` to identify empties since in Blender there are empties that can use `bpy.types.Image` as their `obj.data`.
Andrej730