KeyphraseVectorizers

This package was developed during the writing of our PatternRank paper. You can check out the paper here. When using KeyphraseVectorizers or PatternRank in academic papers and theses, please use the BibTeX entry below.

Set of vectorizers that extract keyphrases with part-of-speech patterns from a collection of text documents and convert them into a document-keyphrase matrix. A document-keyphrase matrix is a mathematical matrix that describes the frequency of keyphrases that occur in a collection of documents. The matrix rows indicate the text documents and columns indicate the unique keyphrases.

The package contains wrappers of the sklearn.feature_extraction.text.CountVectorizer and sklearn.feature_extraction.text.TfidfVectorizer classes. Instead of using n-gram tokens of a pre-defined range, these classes extract keyphrases from text documents using part-of-speech tags to compute document-keyphrase matrices.

Corresponding medium posts can be found here and here.

Benefits

Extract grammatically accurate keyphases based on their part-of-speech tags.
No need to specify n-gram ranges.
Get document-keyphrase matrices.
Multiple language support.
User-defined part-of-speech patterns for keyphrase extraction possible.

How does it work?
Installation
Usage
Citation information

How does it work?

First, the document texts are annotated with spaCy part-of-speech tags. A list of all possible spaCy part-of-speech tags for different languages is linked here. The annotation requires passing the spaCy pipeline of the corresponding language to the vectorizer with the spacy_pipeline parameter.

Second, words are extracted from the document texts whose part-of-speech tags match the regex pattern defined in the pos_pattern parameter. The keyphrases are a list of unique words extracted from text documents by this method.

Finally, the vectorizers calculate document-keyphrase matrices.

Installation

pip install keyphrase-vectorizers

Usage

For detailed information visit the API Guide.

KeyphraseCountVectorizer

Back to Table of Contents

English language

from keyphrase_vectorizers import KeyphraseCountVectorizer

docs = ["""Supervised learning is the machine learning task of learning a function that
         maps an input to an output based on example input-output pairs. It infers a
         function from labeled training data consisting of a set of training examples.
         In supervised learning, each example is a pair consisting of an input object
         (typically a vector) and a desired output value (also called the supervisory signal). 
         A supervised learning algorithm analyzes the training data and produces an inferred function, 
         which can be used for mapping new examples. An optimal scenario will allow for the 
         algorithm to correctly determine the class labels for unseen instances. This requires 
         the learning algorithm to generalize from the training data to unseen situations in a 
         'reasonable' way (see inductive bias).""", 
             
        """Keywords are defined as phrases that capture the main topics discussed in a document. 
        As they offer a brief yet precise summary of document content, they can be utilized for various applications. 
        In an information retrieval environment, they serve as an indication of document relevance for users, as the list 
        of keywords can quickly help to determine whether a given document is relevant to their interest. 
        As keywords reflect a document's main topics, they can be utilized to classify documents into groups 
        by measuring the overlap between the keywords assigned to them. Keywords are also used proactively 
        in information retrieval."""]
        
# Init default vectorizer.
vectorizer = KeyphraseCountVectorizer()

# Print parameters
print(vectorizer.get_params())
>>> {'binary': False, 'dtype': <class 'numpy.int64'>, 'lowercase': True, 'max_df': None, 'min_df': None, 'pos_pattern': '<J.*>*<N.*>+', 'spacy_exclude': ['parser', 'attribute_ruler', 'lemmatizer', 'ner'], 'spacy_pipeline': 'en_core_web_sm', 'stop_words': 'english', 'workers': 1}

By default, the vectorizer is initialized for the English language. That means, an English spacy_pipeline is specified, English stop_words are removed, and the pos_pattern extracts keywords that have 0 or more adjectives, followed by 1 or more nouns using the English spaCy part-of-speech tags. In addition, the spaCy pipeline components ['parser', 'attribute_ruler', 'lemmatizer', 'ner'] are excluded by default to increase efficiency. If you choose a different spacy_pipeline, you may have to exclude/include different pipeline components using the spacy_exclude parameter for the spaCy POS tagger to work properly.

# After initializing the vectorizer, it can be fitted
# to learn the keyphrases from the text documents.
vectorizer.fit(docs)

# After learning the keyphrases, they can be returned.
keyphrases = vectorizer.get_feature_names_out()

print(keyphrases)
>>> ['users' 'main topics' 'learning algorithm' 'overlap' 'documents' 'output'
 'keywords' 'precise summary' 'new examples' 'training data' 'input'
 'document content' 'training examples' 'unseen instances'
 'optimal scenario' 'document' 'task' 'supervised learning algorithm'
 'example' 'interest' 'function' 'example input' 'various applications'
 'unseen situations' 'phrases' 'indication' 'inductive bias'
 'supervisory signal' 'document relevance' 'information retrieval' 'set'
 'input object' 'groups' 'output value' 'list' 'learning' 'output pairs'
 'pair' 'class labels' 'supervised learning' 'machine'
 'information retrieval environment' 'algorithm' 'vector' 'way']

# After fitting, the vectorizer can transform the documents 
# to a document-keyphrase matrix.
# Matrix rows indicate the documents and columns indicate the unique keyphrases.
# Each cell represents the count.
document_keyphrase_matrix = vectorizer.transform(docs).toarray()

print(document_keyphrase_matrix)
>>> [[0 0 2 0 0 3 0 0 1 3 3 0 1 1 1 0 1 1 2 0 3 1 0 1 0 0 1 1 0 0 1 1 0 1 0 6
  1 1 1 3 1 0 3 1 1]
 [1 2 0 1 1 0 5 1 0 0 0 1 0 0 0 5 0 0 0 1 0 0 1 0 1 1 0 0 1 2 0 0 1 0 1 0
  0 0 0 0 0 1 0 0 0]]

# Fit and transform can also be executed in one step, 
# which is more efficient. 
document_keyphrase_matrix = vectorizer.fit_transform(docs).toarray()

print(document_keyphrase_matrix)
>>> [[0 0 2 0 0 3 0 0 1 3 3 0 1 1 1 0 1 1 2 0 3 1 0 1 0 0 1 1 0 0 1 1 0 1 0 6
  1 1 1 3 1 0 3 1 1]
 [1 2 0 1 1 0 5 1 0 0 0 1 0 0 0 5 0 0 0 1 0 0 1 0 1 1 0 0 1 2 0 0 1 0 1 0
  0 0 0 0 0 1 0 0 0]]

Other languages

Back to Table of Contents

german_docs = ["""Goethe stammte aus einer angesehenen bürgerlichen Familie. 
                Sein Großvater mütterlicherseits war als Stadtschultheiß höchster Justizbeamter der Stadt Frankfurt, 
                sein Vater Doktor der Rechte und Kaiserlicher Rat. Er und seine Schwester Cornelia erfuhren eine aufwendige 
                Ausbildung durch Hauslehrer. Dem Wunsch seines Vaters folgend, studierte Goethe in Leipzig und Straßburg 
                Rechtswissenschaft und war danach als Advokat in Wetzlar und Frankfurt tätig. 
                Gleichzeitig folgte er seiner Neigung zur Dichtkunst.""",
              
               """Friedrich Schiller wurde als zweites Kind des Offiziers, Wundarztes und Leiters der Hofgärtnerei in 
               Marbach am Neckar Johann Kaspar Schiller und dessen Ehefrau Elisabetha Dorothea Schiller, geb. Kodweiß, 
               die Tochter eines Wirtes und Bäckers war, 1759 in Marbach am Neckar geboren
               """]
# Init vectorizer for the german language
vectorizer = KeyphraseCountVectorizer(spacy_pipeline='de_core_news_sm', pos_pattern='<ADJ.*>*<N.*>+', stop_words='german')

The German spacy_pipeline is specified and German stop_words are removed. Because the German spaCy part-of-speech tags differ from the English ones, the pos_pattern parameter is also customized. The regex pattern <ADJ.*>*<N.*>+ extracts keywords that have 0 or more adjectives, followed by 1 or more nouns using the German spaCy part-of-speech tags.

Attention! The spaCy pipeline components ['parser', 'attribute_ruler', 'lemmatizer', 'ner'] are excluded by default to increase efficiency. If you choose a different spacy_pipeline, you may have to exclude/include different pipeline components using the spacy_exclude parameter for the spaCy POS tagger to work properly.

KeyphraseTfidfVectorizer

Back to Table of Contents

The KeyphraseTfidfVectorizer has the same function calls and features as the KeyphraseCountVectorizer. The only difference is, that document-keyphrase matrix cells represent tf or tf-idf values, depending on the parameter settings, instead of counts.

from keyphrase_vectorizers import KeyphraseTfidfVectorizer

docs = ["""Supervised learning is the machine learning task of learning a function that
         maps an input to an output based on example input-output pairs. It infers a
         function from labeled training data consisting of a set of training examples.
         In supervised learning, each example is a pair consisting of an input object
         (typically a vector) and a desired output value (also called the supervisory signal). 
         A supervised learning algorithm analyzes the training data and produces an inferred function, 
         which can be used for mapping new examples. An optimal scenario will allow for the 
         algorithm to correctly determine the class labels for unseen instances. This requires 
         the learning algorithm to generalize from the training data to unseen situations in a 
         'reasonable' way (see inductive bias).""", 
             
        """Keywords are defined as phrases that capture the main topics discussed in a document. 
        As they offer a brief yet precise summary of document content, they can be utilized for various applications. 
        In an information retrieval environment, they serve as an indication of document relevance for users, as the list 
        of keywords can quickly help to determine whether a given document is relevant to their interest. 
        As keywords reflect a document's main topics, they can be utilized to classify documents into groups 
        by measuring the overlap between the keywords assigned to them. Keywords are also used proactively 
        in information retrieval."""]
        
# Init default vectorizer for the English language that computes tf-idf values
vectorizer = KeyphraseTfidfVectorizer()

# Print parameters
print(vectorizer.get_params())
>>> {'binary': False, 'custom_pos_tagger': None, 'decay': None, 'delete_min_df': None, 'dtype': <


class 'numpy.int64'>, 'lowercase': True, 'max_df': None

, 'min_df': None, 'pos_pattern': '<J.*>*<N.*>+', 'spacy_exclude': ['parser', 'attribute_ruler', 'lemmatizer', 'ner',
                                                                   'textcat'], 'spacy_pipeline': 'en_core_web_sm', 'stop_words': 'english', 'workers': 1}

To calculate tf values instead, set use_idf=False.

# Fit and transform to document-keyphrase matrix.
document_keyphrase_matrix = vectorizer.fit_transform(docs).toarray()

print(document_keyphrase_matrix)
>>> [[0.         0.         0.09245003 0.09245003 0.09245003 0.09245003
  0.2773501  0.09245003 0.2773501  0.2773501  0.09245003 0.
  0.         0.09245003 0.         0.2773501  0.09245003 0.09245003
  0.         0.09245003 0.09245003 0.09245003 0.09245003 0.09245003
  0.5547002  0.         0.         0.09245003 0.09245003 0.
  0.2773501  0.18490007 0.09245003 0.         0.2773501  0.
  0.         0.09245003 0.         0.09245003 0.         0.
  0.         0.18490007 0.        ]
 [0.11867817 0.11867817 0.         0.         0.         0.
  0.         0.         0.         0.         0.