Create an API module that allows various types of sensors to be connected to FarmOS.

The main goal is to create a really flexible system that allows FarmOS to connect to various types of sensors. I'm running a bunch of Atmospis around my house (http://github.com/mstenta/atmospi), so I'm probably going to whip up something that pulls data from them... and it would be awesome to do it in such a way that Fidos could also be queried (http://farmhack.net/tools/fido-temperature-alarm-sends-text-messages)... and others!

It's tricky because every sensor has it's own way of doing things... so I want to figure out a simple, but flexible, FarmOS module that others can build upon. Maybe it could provide a hook that other modules (ie: farm_fido, farm_atmospi, etc) could implement to declare how to connect to and query themselves.

Individual sensors can be represented as Farm Asset entities, and could provide a "Sensor Type" field that defines the type, along with specific configuration for it (how to connect, etc).

Requirements:

  • General purpose module that allows many types of sensors to be integrated (ie: Fido, Apitronics, Atmospi, custom, etc)
  • Assign sensors to specific areas (via Movement logs).
  • Pull most recent sensor reading(s).
  • Display most recent sensor readings in Area Details popup. (Depends on #2389497: Area details in Openlayers popup)
  • Query time series ranges of data.
  • Lazy load sensor data on demand in UI via Javascript.
  • Provide a custom hook that defines sensor types, and how to communicate with them.
  • ...

Comments

m.stenta’s picture

Issue summary: View changes
BruceDawson’s picture

Most sensors can generate a log. And you can probably create hooks to rules to get them to do special things when certain log entries occur.

Now, actuators are a different matter.

R.J. Steinert’s picture

Sounds like I need to brush the dust off the ol' sensor_hub module. https://www.drupal.org/project/sensor_hub

One of the lessons learned on that project was how slow and memory intensive using nodes for sensor readings can be. Switching to plain tables and using the Data module (https://www.drupal.org/project/data) to let Views know about them was a win.

R.J. Steinert’s picture

MVP for this could be "ability to create a new Device, attach a Sensor" and then "Access to an HTTP API for the Device that populates tables of data that relate to each sensor". Those two user stories are not too hard, but... They may be affected by how to best accommodate a "Browse sensor data" user story. As I mentioned, Data module was good for hooking arbitrary tables up so Views could query them. Is Data good enough for the UI side?

R.J. Steinert’s picture

Some tasks I've been thinking about to accomplish this...

- Create Device Entity
- Create Sensor Entity (relates to Device)
- Utilize node hooks for creating tables for each Sensor
- Create hook_menu entry for POSTing data to a Sensor's table
- Create hook_menu entry for GETing data from a Sensor's table
- Create JS App for browsing a Sensor(s) data
- Encapsulate JS App into a special Headless Block that can feed the right contextual variables to the JS App given the Farm OS environment.

m.stenta’s picture

What if we just set up something relatively simple, and then provided hooks for other modules to use?

Here's what I'm thinking:

New module: Farm Sensor - provides a new asset type called "Sensor". Sensors can be assigned to any asset in the system (including Areas (once this is done: #2363393: Make "Area" into a type of Farm Asset), Compost Piles (#2393103: Compost module), etc (even Sensors can have Sensors!)

Sensors are each assigned a specific "type". The available types should come from other modules, via a new hook_farm_sensor_type().

Each type should also define sensor-specific settings that are needed for that type. For example: the URL of the sensor's data feed, or other stuff that is specific to the actual sensor you're setting up.

So, for example, you could write a Fido module that implements the hook like this:


/**
 * Implements hook_farm_sensor_type().
 */
function fido_farm_sensor_type() {
  $types = array(
    'fido' => array(
      'label' => t('Fido'),
      'settings' => array(
        // ...
      ),
    ),
  );
  return $types;
}

To start, the Farm Sensor module could ask other modules for a simple "Summary" output of the sensor - this could be "Latest Temperature", or it could be something more complicated. There could also be a way to link to a more detailed display of the data, either within Drupal, or offsite (within the sensor's web UI itself, if it provides one).

With an approach like this, we don't have to figure out how to store sensor data within Drupal - just how to communicate in a standardized way with remote sensors or devices. It does assume that the sensor/device itself will have some way to communicate... which isn't a problem for Fido or Atmospi. So maybe those should be the first ones we add support for.

R.J. Steinert’s picture

Hi Mike,
It's good to hear your thinking on how the idea of a Sensor can fit within the framework that already exists for Assets. On the idea of storing sensor data offsite, a year ago I might have said that was a good idea but now I'm not so sure. You can no longer create new devices on Cosm; Apitronics never launched a product; data.sparkfun.com is still limited to 50mb for devices; Dathub wants to be the Github of data but hasn't launched yet (crossing my fingers on this one); Cloudant might be an option if they weren't so developer oriented; and countless other proprietary devices make their own sensor->mote->gateway->database->software stack that we probably won't be able to access from Farm OS. At the moment I'm thinking about some combination of what you proposed with the setup of Phant (data.sparkfun.com), and perhaps schemaless. More thinking to do...
- RJ

R.J. Steinert’s picture

Here's what the set up of a new Sensor might look like...

- User selects "create new Sensor"
- User fills out form with the following fields
  - title
  - type 

Instead of requiring code to declare a sensor type, perhaps just a controlled vocabulary that one could add a new type if they wanted to. When the form is submitted, the user is then presented with a public key, private key, and URL to point the sensor's Mote to. A Mote's job to post data is then as simple as...

HTTP GET http://example.com/stream/{public_key}?private_key={private_key}&data=42

This is almost exactly what Phant does, except in Phant you are creating multi-sensor "Streams" where each sensor value has a shared timestamp. The multi-sensor stream approach has particular advantages, like saving room in the database, maybe it's easier for the User to manage... but this is about data model and the UI could be anything we want. It also reduces the HTTP traffic over the wire, which matters sometimes when you want to conserve battery, but how much of a difference it makes, I have my doubts. I think people's impressions are that it makes sense to have multi-sensor streams over single sensor stream, in the software I work on though it seems like single sensor streams is much simpler to maintain. Maybe I just need to think around the problem more and it will start to look elegant in my head.

m.stenta’s picture

I guess I assumed most sensor devices stored their own data on the device - I don't have any experience though, other than the Atmospi software I wrote for my Pi temperature and humidity sensors. Those are sort of self-contained servers+sensors, which store data in a simple SQLite3 database and serve it via a web UI and JSON. Fido is similar, right? But with couchDB?

So, with your points in mind, I guess we shouldn't assume that most devices store data. And maybe creating a simple data storage mechanism makes sense.

I guess we can divide sensor devices into two camps: those that are only concerned with the present moment (don't store logs of data), and those that keep a log. There's no reason we NEED to enforce keeping a log for devices that don't do that already. We can start with just a simple Farm Sensor API module that allows the CURRENT reading of those devices be displayed in the farmOS UI. It doesn't need to store data or show graphs or anything... yet.

Then, after that, we could create a secondary module that provides some basic mechanisms for storing and graphing data... on behalf of devices that don't do that themselves. But I think that should be an add-on, and we shouldn't assume that all devices will use it. For example, I would like to be able to query my Atmospis for historic data from farmOS, but still leave all the data storage up to them. I'm not married to that idea, though, so I could be swayed. :-)

m.stenta’s picture

Just read through your #8 comment... cool, I'll take a look at Phant. It sounds like that's basically what we want to be able to do. I need to wrap my head around the concept of sensor "streams" though... not sure I get that yet. We should do a Hangout sometime and just talk about farmOS Sensors! :-)

m.stenta’s picture

Project: farmOS » Farm Sensor
Status: Active » Closed (fixed)

RJ, Dogi, and I had the chance to discuss a lot of these things at iFARM in New Hampshire this past weekend.

I put together two proof-of-concept modules for it: farm_sensor and farm_sensor_atmospi. I'll be pushing those up to public repos soon...

http://drupal.org/project/farm_sensor
http://github.com/mstenta/farm_sensor_atmospi

I'm going to close this discussion thread, and move it to the Farm Sensor project. I started creating more specific feature requests in that module's queue, so let's continue discussion in those (and create more as needed).

Data storage: #2493225: Data storage
REST API: #2493227: REST API for pushing data
Graph sensor data: #2493511: Graph sensor data

Remember to subscribe to those issues (or the whole project) so that you're notified when people comment!

m.stenta’s picture

Also, Bruce: wanted to point you to this new one... #2497223: Farm Sensor: Alert

My friend Dylan also mentioned integrating actuators... and how it would allow Drupal to finally become self-aware. :-)

m.stenta’s picture

Project: Farm Sensor » farmOS

Moving old issues to the canonical farmOS project for easier searching. Apologies for email spam if anyone is following these. :-)