Image 1: Flow Chart.
VWUDS Flow Chart

Proposed Solution

  • URL format: ows-vwuds-[bundle or user]-[function: info/edit/data]

Remaining Tasks

Image 2: Navigation.

  1. Develop and deploy dh_perms - contact based permissions.
  2. Migrate dh module.
  3. Migrate dh_wsp module.
  4. Migrate om module.
  5. Create Outline of screens
  6. Monthly/Annual data widget from dh_properties class
  7. User Facilities screen or section in Dashboard

  8. All orgs attached to user (contact or eref), facilities attached to orgs

  9. Organization Facilities screen (see Image 3) - indicates reporting status, shows messages (approvals, status change, review needed)
  10. Facility screen (see Image 4) - link to facility attributes/contacts, list MPs and their reporting status
  11. MP Screen - sub-screens, MP attributes, and annual/monthly reporting data entry

    Intakes - ex: http://deq1.bse.vt.edu/d.alpha/ows-vwuds-intake-monthly/58760

    Wells - ex: d.alpha/ows-vwuds-well-monthly/58741

    Transfer Release - d.alpha/ows-vwuds-transfer-monthly/283568/2016

    Transfer Delivery - d.alpha/ows-vwuds-transfer-monthly/283621/
  12. Do final migrations of withdrawal monthly data (only annual and daily summary data has been migrated) - see http://deq1.bse.vt.edu/sifnwiki/index.php/Drupal_hydro_vwuds_measuring_p...
  13. Migrate transfer features (see related issue for data model)

    Transfer MPs: /files/vwuds/vwuds_tfr.tsv

    Conveyance Features: /var/www/html/files/vwuds/vwuds_transfer_feature-sample.tsv and files/vwuds/vwuds_transfer_features.tsv

    Transfer Data (release): files/vwuds/vwuds.tw-release.tsv

    Transfer Data (delivery): files/vwuds/vwuds.tw-delivery.tsv
  14. Update MPs with fstatus = 8 and 9 to proper value (be sure that "active" is what it should be)
  15. create a view to accept transfer hydroid as input and give a from feature or a to feature entity reference erefid as result for use by string to entity id tamper in feeds import http://deq1.bse.vt.edu/d.alpha/admin/structure/feeds/dh_vwuds_transfer
  16. Review Records where old system had multiple releases to another facility intended for separate service areas underneath the recipient facility -- some cases we may not have created multiple recipient? see table 1: Duplicates.
  17. Investigate the 3 conveyance features that are in the ‘vwuds_transfer_features.tsv’ twice

    vwuds_conv-0282-0288

    vwuds_conv-0283-0476

    vwuds_conv-0232-2039
  18. Import new features from updated NHD impoundment layer.

Image 3: Organization/Owner Status.

Image 4: Facility info page shows status of reporting on associated MPs.

Table 1: Duplicates that maybe should be actual duplicates.


vwuds=# select name, src, dest, hydrocode from vwuds_to_dh_tfr_mps where hydrocode in ('vwuds_tfr-0282-0288-TW-RL', 'vwuds_tfr-0283-0476-TW-DL');
                  name                   | src  | dest |         hydrocode
-----------------------------------------+------+------+---------------------------
 TO PR WILLIAM CO-GMSD                   | 0282 | 0288 | vwuds_tfr-0282-0288-TW-RL
 TO PR WILLIAM CO-GMSD                   | 0282 | 0288 | vwuds_tfr-0282-0288-TW-RL
 TO GMSD SERVICE AREA                    | 0282 | 0288 | vwuds_tfr-0282-0288-TW-RL
 TO GMSD SERVICE AREA                    | 0282 | 0288 | vwuds_tfr-0282-0288-TW-RL
 FROM FAIRFAX CO WA - POTOMAC            | 0283 | 0476 | vwuds_tfr-0283-0476-TW-DL
 FROM FAIRFAX CO WA - SOUTHSIDE - BYPASS | 0283 | 0476 | vwuds_tfr-0283-0476-TW-DL
(6 rows)


Views to Migrate:

  • Done 12/7 - d.alpha/admin/structure/views/view/vwuds_measuring_point_dashboard/edit
  • Done 12/7 - d.alpha/admin/structure/views/view/vwuds_facility_dashboard/edit
  • Done 12/7 - d.alpha/admin/structure/views/view/vwuds_breadcrumbs/edit
  • d.alpha/admin/structure/views/view/vwuds_organization_dashboard/edit
  • d.alpha/admin/structure/views/view/vwuds_organization_page/edit
  • Done 12/7 - d.alpha/admin/structure/views/view/ows_vwuds_mp_status/edit
  • d.alpha/admin/structure/views/view/vwuds_user_dashboard/edit/default
  • d.alpha/admin/structure/views/view/vwuds_hardcopy_template/edit/page
  • d.alpha/admin/structure/views/view/vwuds_facility_mail_merge/edit

Panels to Migrate:

  • Done 12/7 - d.alpha/admin/structure/pages/edit/page-vwuds_facility_dashboard
  • d.alpha/admin/structure/pages/edit/page-vwuds_intake_dashboard
  • Done 12/7 - d.alpha/admin/structure/pages/edit/page-vwuds_measuring_point_dashboard
  • d.alpha/admin/structure/pages/edit/page-vwuds_transfer_dashboard
  • d.alpha/admin/structure/pages/edit/page-ows_vwuds_transfer_monthly
  • d.alpha/admin/structure/pages/edit/page-vwwr_intake_monthly
  • Done 12/7 - d.alpha/admin/structure/pages/edit/page-vwwr_well_monthly
  • d.alpha/admin/structure/pages/edit/page-ows_vwuds_dashboard
  • d.alpha/admin/structure/pages/edit/page-vwuds_facility_contacts
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

robertwb created an issue. See original summary.

robertwb’s picture

Issue summary: View changes
FileSize
43.38 KB
64.09 KB
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Code for forms:

<?php

class DHMonthlyReportingEvent extends dhTimeseriesGroup {
  // @todo:
  //  save handlers
  //  formRowDefaults
  // viewBlock method
  var $event_id = FALSE;
  public function __construct($conf = array()) {
    parent::__construct($conf);
  }
  
  public function entityDefaults() {
    parent::entityDefaults();
    // HEADERS - sets 
    $this->entity_defaults['headers'] = array(
      'tstime' => 'Month', 
    );
    // get default list and order of form columns from blank form
    $blank_form_row = array();
    $this->formRowDefaults($blank_form_row, new StdClass);
    // set form_columns - control WHAT is included (not visibility), and the ORDER
    $this->entity_defaults['form_columns'] = array_keys($blank_form_row);
    // now, we must set headers up here because we do not have a config view 
    foreach ($this->entity_defaults['headers'] as $col => $title) {
      $this->conf['display']['properties'][$col]['title'] = $title;
      if ($blank_form_row[$col]['#type'] == 'hidden') {
        $this->conf['display']['properties'][$col]['hidden'] = 1;
      }
    }
  }
  
  public function entityOptions(&$form, $form_state) {
    parent::entityOptions($form, $form_state);
  }
  
  function prepareQuery() {
    $this->applyEntityTokens();
    $this->applySettings();
    
    $ei = $this->getTSEntityInfo();
    $entity_pkcol = $ei['entity keys']['id'];
    $entity_tbl = $ei['base table'];
    // get app plan adminreg id
    // get linked chems
    // get rate group for each chem
      // varkey = agchem_rate_group
      // propcode = vocab of rate group: agchem_event_lbs, agchem_event_oz
    // get individual components of application
      // datatype = rate
      // datatype = amount
    // iterate through refs
    $q = "  select a.thisyear, b.thismonth, to_timestamp(c.tstime) as tstime, ";
    $q .= "   v.hydroid as varid, c.tid, c.tsvalue as tsvalue0, d.tsvalue as tsvalue1, ";
    $q .= "   e.tsvalue as tsvalue2 ";
    $q .= " from $entity_tbl as mp  ";
    $q .= " left outer join ";
    $q .= " ( ";
    $q .= "    select $this->year::integer as thisyear ";
    $q .= " ) as a ";
    $q .= " on (1 = 1) ";
    $q .= " left outer join (    ";
    $q .= " select generate_series as thismonth from generate_series(1,12) ";
    $q .= " ) as b ";
    $q .= " on (1 = 1) ";
    $q .= " left outer join dh_variabledefinition as v  ";
    $q .= " on (v.varkey = 'wlg') ";
    $q .= " left outer join dh_timeseries as c ";
    $q .= " on ( ";
    $q .= "   c.varid = v.hydroid  ";
    $q .= "     and to_timestamp(c.tstime) = to_timestamp(a.thisyear || '-' || ";
    $q .= " b.thismonth || '-' || 1, 'YYYY-MM-DD' ) ";
    $q .= "     and c.featureid = mp.$entity_pkcol ";
    $q .= "     and c.entity_type = 'dh_feature' ";
    $q .= " ) ";
    $q .= " left outer join dh_timeseries as d ";
    $q .= " on ( ";
    $q .= "   d.varid = v.hydroid ";
    $q .= "     and to_timestamp(d.tstime) = to_timestamp((a.thisyear -1) ";
    $q .= "       || '-' || b.thismonth || '-' || 1, 'YYYY-MM-DD')";
    $q .= "     and d.featureid = mp.$entity_pkcol";
    $q .= "     and d.entity_type = 'dh_feature'";
    $q .= " )";
    $q .= " left outer join dh_timeseries as e";
    $q .= " on (";
    $q .= "   e.varid = v.hydroid ";
    $q .= "     and to_timestamp(e.tstime) = to_timestamp((a.thisyear) - 2 ";
    $q .= "       || '-' || b.thismonth || '-' || 1, 'YYYY-MM-DD')";
    $q .= "     and e.featureid = mp.$entity_pkcol";
    $q .= "     and e.entity_type = 'dh_feature'";
    $q .= " )";
    $q .= " where mp.$entity_pkcol = $this->featureid ";
    $q .= " and v.varkey = '$this->varkey' ";
    $q .= " order by a.thisyear, b.thismonth";
    $this->query = $q;
    return TRUE;
  }
  
  function getData() {
    if (!isset($this->query) or !$this->query) {
      // malformed or non existent query
      return FALSE;
    }
    $this->data = array();
    //get the chems linked to this event
    $q = db_query($this->query);
    dpm($q, "initial data");
    foreach ($q as $prow) {
      // while we go through this we look for duplicates
      // and add an error in the timeseries table for the admin
      // to take a look
       if ($prow->tid == NULL) {
        // this is an insert request
        // not sure if we do anything different between these two cases?
        // maybe check here for default variables?
        //dpm($prow, "Creating blank");
        $this->data[] = $prow;
      } else {
        // not sure if we do anything different between these two cases?
        // maybe check here for default variables?
        $this->data[] = $prow;
      }
    }
  }
  
  public function viewBlock() {
    // this base class does nothing but show block in this->data, all data pop will be done by sub-classes.
    $this->applyEntityTokens();
    $this->applySettings();
    $prop = NULL;
    $rows = array();
    if (!(count($this->data) > 0)) {
      return '';
    }
    $out = '';
    return $out;
  }
  
  public function formRowDefaults(&$rowform, $row) {
    // Row Record:
    //    planid, chemid, name, 
    //    rate_pid, rate_varid, rate_propvalue,
    //    amount_pid, amount_varid, amount_propvalue
    $pc = $this->conf['display']['properties'];
    //dpm($pc, "Prop conf");
    $fc = $this->conf['display']['fields'];
    $rowform['tid'] = array(
      '#type' => 'hidden',
      '#default_value' => $row->tid,
    );
    $rowform['featureid'] = array(
      '#type' => 'textfield',
      '#default_value' => $row->featureid,
    );
    $rowform['entity_type'] = array(
      '#type' => 'textfield',
      '#default_value' => $this->ts_entity_type,
    );
    $rowform['tstime'] = array(
      '#markup' => !empty($row->tstime) ? date('m', $row->tstime) : 'unk',
      '#type' => 'hidden',
      '#default_value' => $row->tstime,
    );
    // static non-dimensional varid
    $rowform['tsvalue'] = array(
      '#type' => 'textfield',
      //'#attributes' => array('disabled' => 'disabled'),
      '#default_value' => empty($row->tsvalue) ? 0.0 : $row->tsvalue,
      '#required' => TRUE,
    );
    $rowform['tsvalue_1'] = array(
      '#markup' => empty($row->tsvalue_1) ? 0.0 : $row->tsvalue_1,
    );
    $rowform['tsvalue_2'] = array(
      '#markup' => empty($row->tsvalue_2) ? 0.0 : $row->tsvalue_2,
    );
    
    $rowform['amount_units'] = array(
      '#markup' => empty($row->units) ? '' : $row->units,
    );
    
    parent::formRowVisibility($rowform, $row);
    
    // need to spoof a form_state for the row to properly load attached fields
    
  }
  
  public function submitForm(array &$form, $form_state) {
    $this->applyEntityTokens();
    $this->applySettings();
    //dpm($form_state,"SubmitForm");
    // identify records that this is supposed to handle
      // main form field name $this->groupname, i.e. ts_group
    // iterate through, do updates for those with PKID, inserts for those with NULL PKID
    
    // *******************
    // DISABLED for now
    return;
    
    foreach ($form_state['values'][$this->groupname] as $record_group) {
      //dpm($record_group,"Record Group - entity type $this->base_entity_type");
      //dpm($this->entity_info,"Entity info");
      // expects rate_pid, rate_value, amount_pid, amount_value for this group
      foreach (array($prefix . '', 'amount') as $prefix) {
        if ($record_group[$prefix . '_pid'] > 0) {
          $e = entity_load_single($this->base_entity_type, $record_group[$prefix . '_pid']);
          $pinfo = entity_get_property_info('dh_properties');
        } else {
          //dpm(1,"Entity created - need to save");
          $rate_group = array();
          $rate_group['varid'] = $record_group[$prefix . '_varid'];
          $rate_group['featureid'] = $this->featureid;
          $rate_group['entity_type'] = $this->prop_entity_type;
          $rate_group['bundle'] = isset($record_group[$prefix . '_bundle']) ? $record_group[$prefix . '_bundle'] : $this->base_bundle;
          $e = entity_create($this->base_entity_type, $rate_group);
          //dpm($e,"Entity created - saving");
          $e->entity_type = $this->prop_entity_type;
          $e->bundle = $this->base_bundle;
        }
        // save the entity
        $e->propvalue = $record_group[$prefix . '_value'];
        $e->save();
      }
    }
  }
  
  public function buildOptionsForm(&$form, $form_state) {
    // Form for configuration when adding to interface
    //   public function buildOptionsForm(&$form, FormStateInterface $form_state) {
    // when we go to D8 this will be relevant
    // until then, we use the old school method
    parent::buildOptionsForm($form, $form_state);
    // we may total over-ride this, but use some of the other guts to do querying
  }
}
?>
c_thomas’s picture

There is an option to use a Select List widget, as well as checklists, in the Flexiform Drupal UI. However, it is not enabled for all field types. I'm not fully familiar with how the field types are set but it seems like that has to do with if you can select the List widget. For example, check out this flexiform I created on d.beta: http://deq1.bse.vt.edu/d.beta/admin/structure/flexiforms/manage/wsp_sour.... You are able to use the Select List widget with both of the entity reference fields (Link to WSP System and VWUDS Source) but none of the others. It looks like this code below would pretty easily let you add any type of form element into a flexiform though, this being checkbox:

$form[$this->element_namespace] = array(
'#type' => 'checkbox',
'#parents' => $parents,
'#title' => $this->label(),
'#default_value' => !empty($entity->status) ? $entity->status : FALSE,
);

robertwb’s picture

Issue summary: View changes
c_thomas’s picture

This is the Drupal page explaining how to add a Flexiform Element: https://www.drupal.org/node/2097817

robertwb’s picture

A sample grid editor can be seen here: http://deq1.bse.vt.edu/d.alpha/node/59/64697

robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
jkleiner’s picture

Used R to create a file of ‘conveyance features’ for import using feeds, complete with line geometries and entity references. The entity IDs of the transfer features need to be used for setting entity references NOT the hydrocodes of those features.

Conveyance Feature Importer: http://deq1.bse.vt.edu/d.alpha/import/vwuds_import_conveyance_feature
Table of conveyance features: http://deq1.bse.vt.edu/d.alpha/vwuds-transfers-import-table?hydrocode
Map of conveyance features: http://deq1.bse.vt.edu/d.alpha/vwuds_transfers_map
Conveyance feature type: http://deq1.bse.vt.edu/d.alpha/admin/structure/dh_feature_type/manage/co...
Table of vwuds features themselves: http://deq1.bse.vt.edu/d.alpha/vwuds-transfers

Outstanding issues resulting in some transfer features not importing:
1. There’s at least one case where the same hydrocode (vwuds_0401) is used for 2 separate features in VAHydro
a. See here: http://deq1.bse.vt.edu/d.alpha/vwuds-transfers?hydrocode_1=vwuds_0401
2. Features that are part of a transfer but DON’T currently exist in VAHydro
3. Features that are part of a transfer and DO exist in VAHydro BUT have bogus geometries (these transfers DO import properly, they just render on a map funky)
a. Ex: -99,99 point location or no lat/long at all

robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes

Added link to export for transfer MPs: http://deq1.bse.vt.edu/files/vwuds/vwuds_tfr.tsv

jkleiner’s picture

Issue summary: View changes
jkleiner’s picture

Issue summary: View changes
jkleiner’s picture

Issue summary: View changes
robertwb’s picture

Debugging these joins:

-- inventory
select foo.userid,  foo.mpid, foo.hydrocode, bar."ACTION", bar."TYPE" from (
 select  'vwuds_tfr_' || "USERID" || '-' || "MPID" as hydrocode,
    "MPID" as mpid,
    "USERID" as userid,
    asText(fac.the_geom) as wkt_geom, count(*)
  from vwuds_measuring_point as mp left outer join facilities as fac
  on (mp."MPID" = fac.userid)
  where "TYPE" = 'TW'
  group by "USERID", "MPID", fac.the_geom
) as foo 
left outer join vwuds_measuring_point as bar 
on (
  foo.mpid = bar."MPID" 
  and foo.userid = bar."USERID"
)
where count > 1;


Results:

 userid | mpid |      hydrocode      | ACTION | TYPE
--------+------+---------------------+--------+------
 0232   | 2039 | vwuds_tfr_0232-2039 | RL     | TW
 0232   | 2039 | vwuds_tfr_0232-2039 | RL     | TW
 0276   | 0283 | vwuds_tfr_0276-0283 | RL     | TW
 0276   | 0283 | vwuds_tfr_0276-0283 | DL     | TW
 0277   | 0282 | vwuds_tfr_0277-0282 | DL     | TW
 0277   | 0282 | vwuds_tfr_0277-0282 | RL     | TW
 0282   | 0288 | vwuds_tfr_0282-0288 | RL     | TW
 0282   | 0288 | vwuds_tfr_0282-0288 | RL     | TW
 0282   | 1462 | vwuds_tfr_0282-1462 | RL     | TW
 0282   | 1462 | vwuds_tfr_0282-1462 | DL     | TW
 0283   | 0276 | vwuds_tfr_0283-0276 | RL     | TW
 0283   | 0276 | vwuds_tfr_0283-0276 | DL     | TW
 0283   | 0277 | vwuds_tfr_0283-0277 | DL     | TW
 0283   | 0277 | vwuds_tfr_0283-0277 | RL     | TW
 0283   | 0469 | vwuds_tfr_0283-0469 | RL     | TW
 0283   | 0469 | vwuds_tfr_0283-0469 | DL     | TW
 0469   | 0282 | vwuds_tfr_0469-0282 | RL     | TW
 0469   | 0282 | vwuds_tfr_0469-0282 | DL     | TW
 0476   | 0283 | vwuds_tfr_0476-0283 | DL     | TW
 0476   | 0283 | vwuds_tfr_0476-0283 | DL     | TW
 0606   | 0711 | vwuds_tfr_0606-0711 | RL     | TW
 0606   | 0711 | vwuds_tfr_0606-0711 | DL     | TW
 0647   | 0785 | vwuds_tfr_0647-0785 | RL     | TW
 0647   | 0785 | vwuds_tfr_0647-0785 | DL     | TW
 0711   | 0606 | vwuds_tfr_0711-0606 | RL     | TW
 0711   | 0606 | vwuds_tfr_0711-0606 | DL     | TW
 1462   | 0282 | vwuds_tfr_1462-0282 | RL     | TW
 1462   | 0282 | vwuds_tfr_1462-0282 | DL     | TW
 1975   | 0248 | vwuds_tfr_1975-0248 | RL     | TW
 1975   | 0248 | vwuds_tfr_1975-0248 | DL     | TW
 2169   | 0300 | vwuds_tfr_2169-0300 | DL     | TW
 2169   | 0300 | vwuds_tfr_2169-0300 | RL     | TW
(32 rows)

robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
jkleiner’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

robertwb’s picture

Issue summary: View changes
FileSize
59.82 KB
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
c_thomas’s picture

Example Facility Dashboard: d.alpha/ows-vwuds-facility-info/66943
Example Owner Dashboard: d.alpha/ows-vwuds-organization-info/65670

robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
FileSize
16.24 KB
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
FileSize
12.19 KB
robertwb’s picture

Added the method, max day columns to the monthly reporting record: http://deq1.bse.vt.edu/d.alpha/ows-vwuds-well-monthly/59840/2016-01-01?d...

robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
c_thomas’s picture

Issue summary: View changes
c_thomas’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
robertwb’s picture

Issue summary: View changes
c_thomas’s picture

Issue summary: View changes
c_thomas’s picture

Issue summary: View changes
c_thomas’s picture

Issue summary: View changes
c_thomas’s picture

Issue summary: View changes