Gender Recognition in Informal and Formal Language Scenarios via Transfer Learning

Abstract

The interest in demographic information retrieval based on text data has increased in the research community because applications have shown success in different sectors such as security, marketing, heath-care, and others. Recognition and identification of demographic traits such as gender, age, location, or personality based on text data can help to improve different marketing strategies. For instance it makes it possible to segment and to personalize offers, thus products and services are exposed to the group of greatest interest. This type of technology has been discussed widely in documents from social media. However, the methods have been poorly studied in data with a more formal structure, where there is no access to emoticons, mentions, and other linguistic phenomena that are only present in social media. This paper proposes the use of recurrent and convolutional neural networks, and a transfer learning strategy for gender recognition in documents that are written in informal and formal languages. Models are tested in two different databases consisting of Tweets and call-center conversations. Accuracies of up to 75% are achieved for both databases. The results also indicate that it is possible to transfer the knowledge from a system trained on a specific type of expressions or idioms such as those typically used in social media into a more formal type of text data, where the amount of data is more scarce and its structure is completely different.

Full text

Gender Recognition in Informal
and Formal Language Scenarios
via Transfer Learning
Daniel Escobar-Grisales1(B
), Juan Camilo V´asquez-Correa1,2,3 ,
and Juan Rafael Orozco-Arroyave1,2
1GITA Lab. Faculty of Engineering, University of Antioquia UdeA,
Medell´ın, Colombia
{daniel.esobar,jcamilo.vasquez,rafael.orozco}@udea.edu.co
2Pattern Recognition Lab., Friedrich-Alexander-Universit¨at,
Erlangen-N¨urnberg, Germany
3Pratech Group, Medell´ın, Colombia
Abstract. The interest in demographic information retrieval based on
text data has increased in the research community because applications
have shown success in different sectors such as security, marketing, heath-
care, and others. Recognition and identification of demographic traits
such as gender, age, location, or personality based on text data can help
to improve different marketing strategies. For instance it makes it pos-
sible to segment and to personalize offers, thus products and services
are exposed to the group of greatest interest. This type of technology
has been discussed widely in documents from social media. However, the
methods have been poorly studied in data with a more formal structure,
where there is no access to emoticons, mentions, and other linguistic
phenomena that are only present in social media. This paper proposes
the use of recurrent and convolutional neural networks, and a transfer
learning strategy for gender recognition in documents that are written
in informal and formal languages. Models are tested in two different
databases consisting of Tweets and call-center conversations. Accuracies
of up to 75% are achieved for both databases. The results also indicate
that it is possible to transfer the knowledge from a system trained on
a specific type of expressions or idioms such as those typically used in
social media into a more formal type of text data, where the amount of
data is more scarce and its structure is completely different.
Keywords: Demographic information retrieval ·Gender recognition ·
Transfer learning ·Recurrent neural networks ·Convolutional neural
networks
1 Introduction
Demographic information retrieval consists in recognizing traits from a human
being such as age, gender, personality, emotions, and others. Typically the main
Supported by University of Antioquia.
c
Springer Nature Switzerland AG 2021
J. C. Figueroa-Garc´ıa et al. (Eds.): WEA 2021, CCIS 1431, pp. 171–179, 2021.
https://doi.org/10.1007/978-3-030-86702-7_15

172 D. Escobar-Grisales et al.
aim is to create a user profile based on unstructured data. The retrieval of
such information has different applications in forensics, security, sales, market-
ing, health-care, and many other sectors [9]. In e-commerce scenarios, this type
of information provides advantages to companies in competitive environments
because it allows to segment customers in order to offer personalized products
and services which strengths their marketing strategies [4,8]. Although most of
the demographic factors are explicitly collected through the registration process,
this approach could be limited given that most of potential customers in online
stores are anonymous. The automatic recognition of demographic variables such
as gender can help to overcome these limitations [5].
Text data from customers can be obtained via transliterations of voice
recordings, chats, surveys, and social media. These text resources can be pro-
cessed to automatically recognize the gender of the users. Different studies
have applied Natural Language Processing (NLP) techniques for gender recog-
nition in text data, mainly from social media posts. In [12,13] the authors used
Term Frequency-Inverse Document Frequency (TF-IDF) to extract features from
tweets in the PAN17 corpus [16], and reported accuracies for gender classification
around 81%. The authors in [3] used extracted features from TF-IDF as well as
specific information only available in social media posts such as the frequency
of female- and male-emojis. The authors reported an accuracy of 83.2% in the
PAN17 [16] corpus for gender recognition. Although the high accuracy reported
in the study, the methodology would not be accurate to model text data writ-
ten in more formal scenarios such as customer reviews, product surveys, opinion
posts, and customer service chats, which have a different structure compared to
the texts that can be found in social media data. In other study [11], the authors
proposed a system to classify the gender of the persons who wrote 100,000 posts
from Weibo (Chinese social network similar to Tweeter). The system was based
on a Word2Vec model, which achieved an accuracy of 62.9%. The authors com-
pared the performance of their model with human judgments, which accuracy
was 60%. This fact evidences that the problem of recognizing gender in written
texts is very hard even for human readers. Wod2Vec models were also consid-
ered in [1] for gender recognition in the PAN17 corpus. The authors reported an
accuracy of 69.5% for the Tweets in Spanish. There are some studies focused on
gender classification using Deep Learning (DL) methods. However, when consid-
ering texts in Spanish, the number of studies is relatively small [9,11]. In [10], the
authors proposed a methodology based on Bidirectional Gated Recurrent Units
(GRUs) and an attention mechanism for gender classification in the PAN17
corpus. The authors worked with a Word2Vec model as input for their DL archi-
tecture and reported accuracies of up to 75.3%.
According to the reviewed literature, gender classification based on text data
has been mainly explored in social media scenarios, where the language is infor-
mal and the documents do not follow a formal structure [6]. These types of doc-
uments use a number of language variants, styles, and other content like emojis
that help to accurately recognize different demographic information. There is a
gap between models trained on formal and informal written language because a

Gender Recognition in Text Data via Transfer Learning 173
trained model with formal language data for a specific purpose will not achieve
comparable results on an informal language scenario, or vice-versa [7]. Due to
this reason, it is important to validate trained models for gender recognition in
both types of languages: formal and informal. In addition, the recognition of
demographic variables such as gender are under-explored in documents with a
more formal structure.
This paper proposes a methodology based on Recurrent Neural Networks
(RNNs) and Convolutional Neural Networks (CNNs) for gender recognition in
informal and formal language scenarios. First, the models are trained and tested
in the PAN17 corpus, which is a traditional dataset for gender classification in
Tweets. The models originally trained using the PAN17 corpus are re-trained
using a transfer learning strategy with data from call-center conversations, which
are structured in a more formal language. Accuracies of up to 75% are obtained,
indicating that the proposed methodology is accurate for gender classification in
documents written in formal and also in informal languages. Moreover, fine-tuned
models using transfer learning show that despite the noise and lack of structure
in documents written in informal language, they can be used to improve the
accuracy of gender classification.
2 Materials and Methods
2.1 Data
PAN17: We are particularly working with the Spanish data of the corpus,
namely PAN-CLEF 2017 [16]. In this database, there are variants of Spanish from
seven countries: Argentina, Chile, Colombia, Mexico, Peru, Spain and Venezuela.
The training set is composed by texts from 600 subjects from each country (300
female). Since each subject has 100 Tweets, there is a total of 4200 subjects and
420000 Tweets in the dataset. The test set comprises data from 400 subjects
from each country (200 female) for a total of 2800 subjects and 280000 Tweets.
For the sake of comparison with previous studies, we kept the original train and
test sets. The training set was randomly divided into 80% for training and 20%
to optimize the hyper-parameters of the models (development set). All data
distribution was performed subject independent to avoid subject specific bias
and to guarantee a better generalization capability of the models.
Call Center Conversations: This corpus contains transliterations of conver-
sations between customers and agents from a customer service center of a pension
administration company in Colombia. Texts are manually generated by a group
of linguistic experts based on the audio signals from the customers. Similarly,
the label of the gender is assigned based on the audio recordings processed by
the linguists. Formal language is typically used by the customers when asking for
a service, making a request, asking about certificates, and other questions about
the service provided by the company. This database comprises 220 translitera-
tions of different customers (110 female). The average number of words for each
conversation is 602, with a standard deviation of 554.

174 D. Escobar-Grisales et al.
2.2 Deep Learning Architectures for Gender Classification
We consider two DL architectures in this case: an RNN with Bidirectional Long
Short Term Memory (LSTM) cells, and a CNN with multiple temporal reso-
lutions. These networks are trained with data from the PAN17 corpus. Then,
a transfer learning strategy is applied to recognize gender from the call center
conversations data.
Bidirectional Long Short Term Memory: The main idea of RNNs is to
model a sequence of feature vectors based on the assumption that the output
depends on the input features at the present time-step and on the output at the
previous time-step. Conventional RNNs have a causa l structure, i.e., the out-
put at the present time step only contains information from the past. However,
many applications require information from the future [15]. Bidirectional RNNs
are created to address such a requirement by combing a layer that processes
the input sequence forward through time with an additional layer that moves
backwards the input sequence. Traditional RNNs also exhibit a vanishing gra-
dient problem, which appears when modeling long temporal sequences. LSTM
layers were proposed to solve this vanishing gradient problem by the inclusion
of a long-term memory to produce paths where the gradient can flow for long
duration sequences such as sentences of a Tweet, or the ones that appear in a
conversation with a call-center agent [17]. We proposed the use of a Bidirec-
tional LSTM (Bi-LSTM) network for our application. These architectures are
widely used for different NLP tasks such as sentiment analysis in social media
and product reviews [2,14,18]. A scheme of the considered architecture is shown
in Fig. 1.
LSTM
Cell
LSTM
Cell
LSTM
Cell
LSTM
Cell
LSTM
Cell
LSTM
Cell
LSTM
Cell
LSTM
Cell
This is aTweet
Embedding
Layer
Bidirectional
LSTM
Clasification
Female Male
Fig. 1. Bi-LSTM architecture for gender classification in a Tweet.

Gender Recognition in Text Data via Transfer Learning 175
Words from the data are represented using a word-embedding layer. The
input to the Bi-LSTM layer consists of kd-dimensional words-embedding vec-
tors, where kis the length of the sequence. The final decision about the gender
of the subject is made at the output layer by using Softmax activation function.
Convolutional Neural Network (CNN): CNN-based architectures are
designed to extract sentence representations by a composition of convolutional
layers and a max-pooling operation over all resulting feature maps. We proposed
the use of a parallel CNN architecture with different filter orders to exploit dif-
ferent temporal resolutions at the same time. Details of the architecture can be
found in Fig. 2. The output from the word-embedding layer is convolved with
filters of different orders (n) and that correspond to different number of the n
in n-grams. The proposed CNN computes the convolution only in the temporal
dimension. After convolution, a max-pooling operation is applied. Finally, a fully
connected layer is used for classification using a Softmax activation function.
Word-embeddings
from document
Convolutional
layer Max-pooling Classification
Four-gram
Tri-gram
Bi-gram
Fig. 2. CNN architecture for gender classification in a Tweet.
2.3 Training
The networks considered in this work are implemented in Tensorflow 2.0, and
are trained with a sparse categorical cross-entropy loss function using an Adam
optimizer. An early stopping strategy is used to stop training when validation
loss does not improve after 10 epochs. The embedding dimension dis set to 100.
The vocabulary size for the tokenizer is set to 5000 for the PAN17 corpus and
1500 for the call-center conversations. The difference between the two vocabulary
sizes is given because the number of unique words present in the training sets
of each database. Hyper-parameters are optimized upon the validation accuracy
and the simplest model.

176 D. Escobar-Grisales et al.
2.4 Transfer Learning
We tested two approaches for the call-center conversations data: (1) training the
network only using the data from the corresponding corpus, and (2) training
the model via transfer learning by using a pre-trained model generated with
the PAN17 corpus. For the transfer learning approach, the most accurate model
for the PAN17 data is fine-tuned, but freezing the embedding layer in order to
keep the tokenizer and a bigger vocabulary. Experiments without freezing the
embedding layer were also performed but the results were not satisfactory. The
motivation for using transfer learning is to test whether the knowledge learned by
a model trained with text data in informal language is useful to improve gender
classification systems based on text with formal language, since it is generally
common to collect large amounts of data with informal structure through social
media, but it is difficult to collect written documents with a formal structure.
3 Experiment and Results
Two experiments are performed in this study. The first one consists in evaluating
short sequences of texts, so the architectures are trained and the gender of the
subject is computed based on the average classification scores of all short texts
from the same subject. Note that for PAN17 corpus, each tweet is a short text,
while for call-center transliterations each conversation is divided into chunks
with 60 words, similar to the proposed in [10]. The second experiment consists
in evaluating long assessment of texts. In this case, the complete text data from
the subjects is entered to the network at the same time. For the PAN17 corpus all
Tweets are concatenated, and for the call-center conversations we consider the
complete transliteration of each conversation. This strategy is only evaluated
using the CNN-based approach because longer segments produced vanishing
gradient problems in the Bi-LSTM network.
Experiments with PAN17: The results obtained for the PAN17 corpus (infor-
mal language) are shown in Table 1for both approaches, short and long texts
evaluation. Bi-LSTM and CNN networks are considered. The best results are
obtained using the strategy with long texts in the CNN. There is an improve-
ment of up to 4% in the accuracy per subject with respect to the accuracy
obtained with short texts. The improvement in the F1-score is around 2%.
Experiments with Call-Center Conversations: The results observed for
the call center conversations (formal language) are shown in Table2. The results
include those obtained with and without applying transfer learning. The results
also include the ones obtained using short and long texts. The results for this
corpus are obtained following a 10-fold cross-validation strategy due to the small
size of the corpus. The highest accuracy is obtained here also with the long texts,
similar to the results obtained with the PAN17 corpus. In addition, note that the
accuracy improves in up to 20% when the transfer learning strategy is applied

Gender Recognition in Text Data via Transfer Learning 177
Table 1. Results of the gender classification in the PAN-CLEF 2017 database.
Short texts Long texts
Bi-LSTM CNN CNN
Accuracy per Tweet 60.5 61.1 –
Accuracy per subject 71.3 71.9 75.9
Precision 68.6 81.1 75.6
Recall 72.0 68.0 76.1
F1-Score 70.8 73.9 75.8
with respect to the accuracy obtained without using the pre-trained models.
Note also that models using transfer learning show a smaller standard deviation
which likely indicates that these methods are more stable.
Table 2. Results of the gender classification in the call-center conversations data. TL:
transfer learning.
Short texts Long texts
Bi-LSTM
Without TL
Bi-LSTM
With TL
CNN
Without TL
CNN
With TL
CNN
Without TL
CNN
With TL
Accuracy per text 52.7 ±6.43 51.6 ±5.07 57.9 ±9.20 58.3 ±6.48 – –
Accuracy per subject 54.2 ±10.1 56.4 ±12.1 65.9 ±12.7 62.9 ±14.9 55.9 ±11.9 75.0 ±6.18
Precision 65.3 ±29.2 55.0 ±13.8 52.0 ±22.2 61.1 ±17.9 54.6 ±25.4 77.2 ±8.12
Recall 53.3 ±22.7 56.7 ±13.2 70.4 ±17.2 64.6 ±15.2 54.9 ±34.8 72.1 ±10.4
F1-Score 55.2 ±19.8 55.2 ±11.9 57.8 ±19.9 61.1 ±15.7 48.5 ±24.0 73.8 ±6.06
4 Conclusions
We proposed a methodology for automatic gender classification based on formal
texts such as those available in social media posts, and based on formal texts
collected in call center conversations. Different deep learning models are eval-
uated including one on Bi-LSTMs, another one based on CNNs and a transfer
learning approach, which is pre-trained with data collected from social networks.
The transfer learning method is fine-tuned to improve the accuracy of the model
designed for text classification in formal languages. The results indicate that
it is possible to classify the gender of a person based on his/her written texts
with accuracies of about 75% in informal and formal language scenarios. The
use of a transfer learning strategy improved the accuracy in scenarios where it
is more difficult to collect data like in call-center conversations, indicating that
this strategy is suitable for companies or sectors where it is not possible to create
large datasets from scratch. The models using transfer learning are also more
stable and generalize better than others where the neural networks are trained

178 D. Escobar-Grisales et al.
from scratch. This is very positive since it is possible to benefit from the large
amounts of text data that are available in other domains like the social net-
works. The proposed approaches can be extended to other applications related
to demographic information retrieval such as age recognition, geographic loca-
tion, personality of the subjects, and others, which would allow the building of
more complete and specific author/customer profiles.
Acknowledgments. This work was funded by the company Pratech Group S.A.S
and the University of Antioquia, grant # PI2019-24110. We would like to thank the
Natural Language Engineering Laboratory of the Universidad Polit´ecnica de Valencia
for providing access to one of the databases used in this work.
References
1. Akhtyamova, L., Cardiff, J., Ignatov, A.: Twitter author profiling using word
embeddings and logistic regression. In: Conference and Labs of the Evaluation
Forum (CLEF) (Working Notes) (2017)
2. Arras, L., Montavon, G., M¨uller, K.R., Samek, W.: Explaining recurrent neural
network predictions in sentiment analysis. arXiv:1706.07206 (2017)
3. Basile, A., et al.: N-gram: new groningen author-profiling model. arXiv:1707.03764
(2017)
4. Dogan, O., Oztaysi, B.: Gender prediction from classified indoor customer paths by
fuzzy C-medoids clustering. In: Kahraman, C., Cebi, S., Cevik Onar, S., Oztaysi,
B., Tolga, A.C., Sari, I.U. (eds.) INFUS 2019. AISC, vol. 1029, pp. 160–169.
Springer, Cham (2020). https://doi.org/10.1007/978-3-030-23756-1 21
5. Fernandez-Lanvin, D., de Andres-Suarez, J., Gonzalez-Rodriguez, M., Pariente-
Martinez, B.: The dimension of age and gender as user model demographic factors
for automatic personalization in e-commerce sites. Comput. Stand. Interfaces 59,
1–9 (2018)
6. Gonz´alez Berm´udez, M.: An analysis of twitter corpora and the differences between
formal and colloquial tweets. In: Proceedings of the Tweet Translation Workshop
2015, pp. 1–7. CEUR-WS.org (2015)
7. Gu, J., Yu, Z.: Data annealing for informal language understanding tasks.
arXiv:2004.13833 (2020)
8. Hirt, R., K¨uhl, N., Satzger, G.: Cognitive computing for customer profiling: meta
classification for gender prediction. Electron. Mark. 29(1), 93–106 (2019). https://
doi.org/10.1007/s12525-019-00336-z
9. Hsieh, F., Dias, R., Paraboni, I.: Author profiling from Facebook corpora. In:
Proceedings of the Eleventh International Conference on Language Resources and
Evaluation (LREC 2018) (2018)
10. Kodiyan, D., et al.: Author profiling with bidirectional RNNs using attention
with GRUs. In: Conference and Labs of the Evaluation Forum (CLEF), vol. 1866.
RWTH Aachen (2017)
11. Li, W., Dickinson, M.: Gender prediction for Chinese social media data. In: Con-
ference Recent Advances in Natural Language Processing (RANLP), pp. 438–445
(2017)
12. Markov, I., G´omez-Adorno, H., Sidorov, G.: Language-and subtask-dependent fea-
ture selection and classifier parameter tuning for author profiling. In: Notebook for
PAN at Conference and Labs of the Evaluation Forum (CLEF) (2017)

Gender Recognition in Text Data via Transfer Learning 179
13. Martinc, M., Skrjanec, I., Zupan, K., Pollak, S.: Pan 2017: author profiling-gender
and language variety prediction. In: Conference and Labs of the Evaluation Forum
(CLEF) (Working Notes) (2017)
14. Minaee, S., Azimi, E., Abdolrashidi, A.: Deep-sentiment: sentiment analysis using
ensemble of CNN and Bi-LSTM models. arXiv:1904.04206 (2019)
15. Otter, D.W., et al.: A survey of the usages of deep learning for natural language pro-
cessing. IEEE Trans. Neural Netw. Learn. Syst. (TNNLS) 32(2), 604–624 (2020)
16. Rangel, F., Rosso, P., Potthast, M., Stein, B.: Overview of the 5th author profiling
task at pan 2017: gender and language variety identification in twitter. In: Working
Notes Papers of the Conference and Labs of the Evaluation Forum (CLEF), pp.
1613–0073 (2017)
17. Torfi, A., Shirvani, R.A., Keneshloo, Y., Tavvaf, N., Fox, E.A.: Natural language
processing advancements by deep learning: a survey. arXiv:2003.01200 (2020)
18. Trofimovich, J.: Comparison of neural network architectures for sentiment analysis
of Russian tweets. In: Computational Linguistics and Intellectual Technologies:
Proceedings of the International Conference “Dialogue”, pp. 50–59 (2016)