CVS edit link for benoit.borrel

Hello,

I am a senior PHP developer with 2 years of experience with Drupal. I had already provided (very) few patches to some modules and would like to contribute to the community my own new module.

My module, Semantic Similarity, computes semantic similarity scores between nodes. To do so, it integrates Drupal with the R Project for Statistical Computing and its Latent Semantic Analysis package. The similarity score (or semantic relatedness: http://en.wikipedia.org/wiki/Semantic_relatedness) is then obtained from a Latent Semantic Analysis algorithm (http://en.wikipedia.org/wiki/Latent_semantic_analysis) which is well established in natural language processing.

Other modules are already listing similar nodes, but from a taxonomy perspective, not from a truly semantic one. That's why I hope my proposal will be accepted as a usefull contibution to the Drupal project.

Thanks for your offering my proposal a chance,

Benoit Borrel

CommentFileSizeAuthor
#8 semantic_similarity-6.x-0.2.tar_.gz33.91 KBbenoit.borrel
#1 semantic_similarity-6.x-0.1.tar_.gz7.68 KBbenoit.borrel

Comments

benoit.borrel’s picture

benoit.borrel commented
StatusFileSize
new semantic_similarity-6.x-0.1.tar_.gz7.68 KB
benoit.borrel’s picture

benoit.borrel commented
Status: Postponed (maintainer needs more info) » Needs review
avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented
Issue tags: +Module review

Hello, and thanks for applying for a CVS account. I am adding the review tags, and some volunteers will review your code, pointing out what needs to be changed.

As per Apply for contributions CVS access, the motivation should be expanded, and include a description about the differences between the proposed module, and the existing ones.

benoit.borrel’s picture

benoit.borrel commented

Thanks for reviewing my proposed module.

As required, here is an expanded motivation message:

My module, Semantic Similarity, automatically computes the semantic similarity score between nodes. On node pages, these scores are then used to display two blocks: Most semantically similar nodes, and Least semantically similar nodes. The former block contains the links to the five most semantically similar nodes and the latter contains the links to the five least similar nodes.

The scores are computed by integrating Drupal with the R Project for Statistical Computing and its Latent Semantic Analysis package. The semantic similarity scores, obtained from a Latent Semantic Analysis (http://en.wikipedia.org/wiki/Latent_semantic_analysis) algorithm, which is well established in natural language processing, is a measure of semantic relatedness (http://en.wikipedia.org/wiki/Semantic_relatedness).

As I stated in Methods to detect relations of similarity between nodes (http://groups.drupal.org/node/45340), many modules offer, based upon different methods, functionalities that serve to detect relation of similarity between nodes (sometimes named "more like this", relevant, similar...). I classified these methods as taxonomy/CCK based and content based.

The first method relies on term-matching between nodes (like module http://drupal.org/project/similarterms) or even let users create complex/custom defined weight and compound associations (like module http://drupal.org/project/Associated_nodes). An (incomplete) list of modules using such method is here: http://drupal.org/node/323329.

The second method relies content-matching between nodes. The only existing module belonging to this method (as far as I know is http://drupal.org/project/similar) relies on MySQL full text searching to perform basic natural language processing.

My proposed module Semantic Similarity, is also using a content based method but utilizes advanced natural language processing. My module offers a truly semantic approach that applies the Latent Semantic Analysis (LSA) algorithm to approximate the meaning of texts, thereby exposing semantic structure to computation. LSA combines the classical vector-space model — well known in computational linguistics — with a singular value decomposition (SVD), a two-mode factor analysis. Thus, bag-of-words representations of texts can be mapped into a modified vector space that is assumed to reflect semantic structure. The module then computes the Pearson correlation coefficient to measure the distance amongst the vectors. This distance is in fact a measure of semantic relatedness between texts.

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented
Status: Needs review » Needs work
  1. See http://drupal.org/coding-standards to understand how a module should be written; in particular, see the part about the namespace respect. 
  2.   // Delete all variables which name begin with semantic_similarity
  $result = db_query("SELECT name FROM {variable} WHERE name LIKE 'semantic_similarity:%%'");
  while ($row = db_fetch_array($result)) {
    variable_del($row['name']);
  }

    Drupal variables should have names containing characters from a to z, numbers, and the underscore character.
    To delete them, it's better to delete them using variable_del(), and avoiding to execute a query similar to the one executed. 

  3. define('SEMANTIC_SIMILARITY_PATH',        file_directory_path() .'/semantic_similarity');
define('SEMANTIC_SIMILARITY_CORPUS_PATH', file_directory_path() .'/semantic_similarity/corpus');
define('SEMANTIC_SIMILARITY_TEXT_PATH',   file_directory_path() .'/semantic_similarity/text');

    Those variables are defined in two files, which could be loaded at the same time; this would cause PHP to return an error about constants already defined. 

  4.   $infile_name = drupal_get_path('module', 'semantic_similarity') . '/R/compute_similarity_scores.R';
  $outfile_name = $infile_name .'out';
  $filename = SEMANTIC_SIMILARITY_PATH .'/scores.csv';
  $cmd = sprintf("R CMD BATCH --slave --no-timing '--args %s %s %s' %s %s 2>&1",
    SEMANTIC_SIMILARITY_CORPUS_PATH, SEMANTIC_SIMILARITY_TEXT_PATH, $filename,
    $infile_name, $outfile_name);

  passthru($cmd, $return_var);

    I am not sure it's possible to execute files that are inside the modules directory (apart PHP files). 

  5.   // Insert new scores into database
  $sql = "INSERT INTO {node_semantic_similarity} (snid, tnid, score) VALUES ";
  foreach ($scores as $tnid => $score) {
    $values[] = sprintf("(%d, %d, %f)", $nid, $tnid, $score);
  }
  $result = db_query($sql . implode(', ', $values));

    This is not how an SQL query should be built, and how the parameters should be passed to the query. 

  6.   // Get nids
  $nids = _get_similar_nids_with_score($nid, $order, $limit);

  // Fetch nids into items
  foreach ($nids as $k => $v) {
    $node = node_load($v['nid']);
    $items[] = sprintf('%0.3f', $v['score']) .' '. l($node->title, $node->path);
  }

    The code doesn't filter out the nodes to which the user doesn't have permission to view them; this is considered a security issue.

benoit.borrel’s picture

benoit.borrel commented

Hi Kiamlaluno,

First of all, great thanks for the review.

Here are my comments:

  1. Do you mean I need to rename my "private" functions following this scheme: _module_name_function_name()?
  2. The ':' was a typo. Anyway, I removed this code because the module do not use system variables yet.
  3. Fixed. I remembered I quickly/dirty coded this way because at hook_install() the file module_name.module was not loaded yet.
  4. I was aware of this possible ACL issue. Anyway, further testing have demonstrated some performance issues and I therefore need to rethink how to integrate R with Drupal. I am postponing my contribution till I come up with an efficient solution.
  5. I know it's not Drupal compliant, but extended insert is way more efficient than several insert statements, specifically when records number = nodes number. By the way, this is not user supplied data here, so my way should be safe.
  6. Fixed.
avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented
Status: Needs work » Closed (won't fix)

There have not been replies in the last week. I am marking this application as won't fix.

benoit.borrel’s picture

benoit.borrel commented
Status: Closed (won't fix) » Needs review
StatusFileSize
new semantic_similarity-6.x-0.2.tar_.gz33.91 KB

Hello,

I am proposing a revised version of my module for which my motivation remain the same.

To follow-up on my last list of items in comment #6:

  1. All my private functions name are now prefixed with _semantic_similarity. Eg: _semantic_similarity_foo().
  2. The module is now using system variables which are properly deleted in my hook_uninstall() implementation.
  3. Still fixed ;-)
  4. ACL issue fixed as well as performance issues.
  5. Still not Drupal compliant but secure and way more effective.
  6. Still fixed ;-)

Thanks to review it!

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented
Status: Needs review » Closed (won't fix)

Please read the following links as this is very important information about CVS applications.

Drupal.org has moved from CVS to Git! This is a very significant change for the Drupal community and for these applications. Please read Migrating from CVS Applications to (Git) Full Project Applications and Applying for permission to opt into security advisory coverage on how this affects and benefits you and the application process. In short, every user has now the permissions necessary to create new projects, but they need to apply for opt into security advisory coverage. Without applying, the projects will have a warning on projects that says:

This project is not covered by Drupal’s security advisory policy.

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented
Component: Miscellaneous » new project application
Issue summary: View changes