Interpreting studies has so far tended to concentrate on simultaneous interpreting over the consecutive mode. Note-taking - an integral part of consecutive interpreting - has therefore received very little scholarly attention. As an indispensable tool in consecutive interpreting, note-taking plays an important role in supporting the interpreter's memory. This study argues, however, that the interpreter's notes should not be viewed merely as a memory storage tool, but as a third visual language with its own logic and meaning-making practices that need interpreting. The way in which interpreters read their notes is explored here from the perspective of Social Semiotics for two reasons. Firstly, Social Semiotics conceptualises signs as meaning-making resources which are realized in specific communicative contexts to convey specific communicative intentions - unlike previous approaches to note-taking, that have tended to categorise signs as static constituents of relatively finite sign codes. Secondly, Social Semiotics not only accounts for how written language is used in notes, but also how the pictorial component of communication is encoded and interpreted through interpreter's notes.The interpreter, as a viewer, has to make use of semiotic resources deployed in the notes in order to reconstruct the information given by the speaker and to produce the target speech for the audience. Therefore, the interpreters' note-reading stage, based on the interaction between signs, can be conceptualised by reference to the concept of visual grammar. This study draws on visual grammar (Kress and van Leeuwen 2006) to analyse interpreter's notes with a view to gaining a better understanding of how linguistic and visual semiotic resources are deployed in the process of note-taking. Insight into interpreters' meaning-making practices and note-taking patterns is gained through an experimental study of the notes produced by nine qualified, practising conference interpreters, during a consecutive interpreting task from English into Chinese. The patterns identified in my data set are then compared with the established prescriptive approaches to note-taking training - which are typically based on relatively stable correspondences between note-taking signs/symbols and their meaning.The analysis focuses on certain elements of the source speech (concepts that can be noted down through the use of vectors, geometrical shapes, specific classificational structures, margin, and salience) as reflected in the notes. The way in which interpreters read their notes involves the interaction between two core modes, such as image and language, and a range of sub-modes, such as vectors, geometrical shapes, composition, framing, salience and calligraphy. The results of the analysis indicate that the way in which interpreters arrange the contents of their notes reflects the depth of the information processing effort required by the note-taking process. The findings suggest that the narrative structure in notes seems to assist interpreters in retrieving information at a micro, lexical level, whereas the visual structure would appear to assist interpreters in retrieving information at a macro, contextual level, e.g. in representing the hierarchies of information value, constructing the structure of rendition, and showing the importance of specific signs.