Author Archives: chad.e - Page 2

KootnetSensors does run on Ubuntu!

It was surprisingly easy to get Kootnet Sensors installed on Ubuntu. After tweaking a few Raspbian specific areas, it installed just fine!

So this is awesome and will help with development, as I can test things on much faster systems before deploying to the Pi Sensors. I can also run it from my development environment now (making sure the original service is disabled). So yay!

I suppose this opens up the possibility of using this on other platforms too, like Windows. I have no plans to support Windows as a sensor but it’s nice to know someone could donate time to add support or fork the project.

I just wanted to say it works on Ubuntu. W00t.

Improvements, Fixes, and a New Server-Side Release

I have been working on the final touches for the past few days. Improving logging, refactoring and finalizing Open Sense Map as best I can. Kootnet Sensors now supports adding the sensor onto an Open Sense Map account with the press of a button. It will automatically add all supported sensors to Open Sense Map but I’m not sure how to get the ID’s required through the API, so you have to log in to Open Sense Map and grab the IDs used in the settings section of Kootnet Sensors for Open Sense Map.

I have already sent Open Sense Map an email asking about how to get those ID’s through the registration process, so hopefully, they’ll reply soon.

Things seem fairly stable right now, so I released Version Alpha.27.126 under the standard update channel.

I’m pondering what to work on next since there are a few area’s I need to update now. Here’s a list of my Todo’s so far.

  1. Continue working on ‘Sensor Control’ – This is a web-based replacement for the PC Control Center program. Right now it can check if up to 20 other sensors are Online.
  2. Update the Control Center program.
    • Support the new Online Service Configuration files.
    • Incorporate improvements made to Graphing from the Sensor Side.
  3. Include the required sensor “Drivers” and other Python modules instead of downloading them – This should speed up installation and standardize the module versions. Some will always be downloaded, such as Flask to ensure security updates.
  4. Update Sensor reports with newly learned CSS & HTML.
  5. Add more sensor support (This one will never be complete, always adding more sensors).

Another thing I have been thinking of doing is adding support to install the Sensor Server side on Ubuntu. I’m not sure I could get the main sensors working with it, but at least I could more quickly view changes I want to test in areas outside the sensors, such as cosmetic changes to the HTML pages… In fact… I have not actually tried to install it on Ubuntu … it might just work now… Guess that’s the first step, see what breaks 🙂

That’s it for today.

Enhancements and more Online Services

I had someone email me about my program recently who asked if I was going to support more online services like Luftdaten. This was great, because I posted a few days before, asking what other Online Services I could add support for. In fact, I still have gotten no replies on the forum post asking about other services. Through Luftdaten, I also found Open Sense Map, which I am adding support for right now.

Luftdaten is already working and supported but Open Sense Map has a much more complicated API with Logins, Tokens, Station ID and even Sensor IDs (one per sensor). It looks like the API is pretty comprehensive, as I can even create new accounts, sensor stations, change sensor settings, etc. So far I have 2 sensors uploading to Open Sense Map with manually set Sensor ID’s and such, so next, I want to be able to just enter in a Username / Password and have the program create the sensor under their account with included Sensor IDs and such (Program use Simplicity is a project goal).

I modified my Graphing to use multiprocessing instead of threading. There was no difference in the time it took to finish a graph, but the web interface seemed more responsive during its creation on the Pi’s with more then 1 CPU core.

Let’s see … what else… I have started on the web portals “Sensor Control”, which allows interaction with multiple sensors (up to 20 right now). Sensor Control is going to be a “replacement interface” for the PC software Control Center. I don’t think I’ll actually get rid of the Control Center, as it can create graphs on much faster computers and interact with offline databases but I will be putting more focus on the Web App’s side of things. So far it can check to see if other sensors are online. Next, I’ll be adding the ability to create system and configuration reports, then the ability to change settings on multiple sensors at a time (Good for the deployment of multiple similar sensors).

That’s it for now.

Added Weather Underground support and version Alpha.26.324 released

I have added a new section in the Web Portal for 3rd party online services. Right now it supports a service called ‘Weather Underground’, which from my understandings is a website that hosts a collection of environmental readings provided by citizen donating the use of their weather stations. To get it working, you need an account with Weather Underground, then you can add a device on their website to get an ID and Key, which is what you need to get it working.

I also created a post about my Sensor Project on the Raspberry Pi forums. I’m hoping to get a bit of feedback.

This marks the final release of Alpha.26.324. Since I’m mostly making changes to the server and HTML side of things, I’m releasing Control Center versions that look like this “Alpha.26.x”, indicating it was released supporting the Alpha.26 branch.

HTTPS Enabled!

I am now using HTTPS instead of HTTP. I even have self-signed certificates generate automatically if they are not present in the ssl_files folder. This allows 3 things.

  1. If you don’t have your own certificates, it creates them when the program launches, creating a user-friendly experience. If they are already in the folder, it simply loads them.
  2. You can replace the certificate files in the ssl_files folder with your own to get the nice green bar in your browser.
  3. Simply delete all the files in the ssl_files folder, reboot the sensor and you have new certificates that are good for 5 years.

Initially, I thought that incorrect time on the sensor would not allow you to connect to the sensor through SSL, but I modified the time by 17 hours and it still connected just fine. So that’s good, as I was worried when the Pi’s reboot or shut down for a bit, the time would drift without internet and you would be unable to configure it after that. I’m glad I was wrong!

Now the sensors are actually decently secure! With HTTP authentication and unique SSL generation per sensor, you can rest assured your commands and logins will not be ‘sniffed’ off the network.

One last thing I wanted to do is redirect any HTTP traffic to HTTPS, but I’m not sure how to make that happen with the webserver module I’m using in Python (gevent’s pywsgi).

I have now started to work on updating documentation since a lot has changed. Once that’s done, I’ll probably release another version.

On another note, I used chromes developer console to see the difference of page delivery sizes with gzip enabled. It really helped on the plotly graph’s, as it cut the download size to 1/4 and since they are usually about 4MBs, it makes a difference.

Tweaking Installer Scripts

Since the sensor software is working fairly well, I’m making sure the installer scripts are where they should be. I want to be able to flash Raspbian, download the installer script, run it and have everything just work. So to that end, I made the following changes.

To start, I added in the CLI commands to enable SPI and i2c in the installer script, along with the rfkill unblock wifi to enable wifi on the Pi Zeros (Didn’t seem necessary on the Pi3B+, just the country code allowed it to work). I have also added a function in the Enviro+ sensor module to enable the serial connection for the PSM5003, if it is enabled. The only downfall of the PSM5003 settings is the system has to reboot before it takes effect. So if you enable the PSM5003, you need to allow the program to start up, then reboot the system. It also leaves a warning message in the primary log to reboot.

Thanks to the above-mentioned additions, you can just flash a fresh Raspbian image, download and run the Sensor installer script and everything starts to work! If you have the desktop version of Raspbian installed, there is even a shortcut to the web config in the menu that points to localhost. If you use the lite version of Raspbian without a desktop, you’ll need to access the web configuration from another device (computer, phone, tablet, etc) through its actual IP address OR run the edit-config script locally (Although the script to edit configurations just opens the text files, so it’s not as nice as the web interface).

Development Updates

I have been spending some time working on the web interface for my sensors. I’m pretty happy with the configuration section now, which no longer has any text editors for updating the configuration, it is all web checkboxes, textboxes, radios, and buttons, which all have validation checks in place to make sure user entries are valid (for the most part).

I have also updated the underlying networking configuration to use dhcpcd.conf instead of interfaces. This has the advantage of being able to set static entries per network interface while allowing the use of any other installed adapters. AKA you can still plug in the Ethernet cord into the Pi3b+ for example and get a DHCP address, even if the wireless is configured with static entries. I had to remove the dual wireless network connections in wpa_supplicant.conf, as certain configurations would kill networking altogether, so… better safe than sorry. I might go back and figure it out later, but it’s not really necessary, so it’s on the back burner.

I fixed a few bugs, like the graphing showing incorrect datestamp correlations if a sensor was added later or disabled temporarily. This was caused by my attempt to re-use the datetime recording column for all sensors because they all use the same column, but if a sensor was added later, there would be a big NULL gap, so the first datetime columns would line up with the first actual data of a sensor column. To fix it, I grab the datetime column based on the presence of each sensor reading. Luckily it didn’t seem to add TOO much extra time to render the graphs.

I also added gzip page compression to Flask, which downs the page transfer sizes decently. This will help if I set up some 900Mhz long-distance transmitters since their data rates are not great.

Other changes in no particular order include

  • Refactoring aspects of the web interface, such as using HTML tabs to go through configurations and log entries (This makes it a lot easier to look over things quickly, instead of having to bounce back to the management page to choose another configuration or log.
  • Adjusted plotly graphs to use a dark theme
  • Created file for cached variables to lessen disk reads
  • Updated Ohms Gas resistance to K Ohms for easier human readability
  • Standardized HTML element colors
  • Added links to many sections to help explain what it is (Sensor readings, for example, take you to the Wiki page for what it is)
  • Other misc. refactoring and bug fixes

I’m getting closer to releasing a beta version, as a lot of the core parts are pretty stable. The networking was also problematic since it used to disable all other networks and was not easily adjustable. Now it works well and allows it to be configured in such a way that the device can still be used online in other ways besides being a sensor.

The web app is coming along nicely, which will eventually overtake the PC Control Center App (I pretty much just have to add the ability to manage multiple sensors from the web app).

That’s it for today’s update.

New Release Ver. Alpha.25.252

Another release! I actually released this one awhile ago, and am already on development version Alpha.26.140, of which I have changed a fair amount again but I should probably start by saying whats new in 25.252.

To start, I have been spending a fair amount of time researching and implementing CSS with a framework called “MUI“. It has some great looking functions to create buttons, tabs, dropdown menus, etc. I’m also slowly figuring out how to adjust multiple HTML components by using style to target classes, names, ids, etc. So that’s coming along nicely and the web interface looks a lot nicer thanks to it. I actually don’t have any sensors running the previous version, so I kinda forget a bunch of what I did, but I’m pretty sure I added a bunch of functions, such as updating to standard or development editions, edit configuration files, reboot/shutdown/restart services and view, download and even clear logs.

In my Development build, I have added the following in no particular order.

  1. New CSS side menu
  2. New Log Viewer page
  3. New Configuration adjustment page
  4. Ability to create and view Plotly graphs from the local sensor webpage
  5. Other visual adjustments, enhancements & bug fixes

I’m still working on improving the look, feel and functionality of the web interface and to that end I’ll be putting in an option for different themes since I know not everyone is a fan of the dark theme I have going.

Another thing I have been working on for a few days is getting the Particulate Matter and GAS sensors working properly for recording and it’s now working fairly well. Since the PM and GAS need to “warm-up”, I created threads to constantly pull data every second to keep it alive for more accurate recordings. It seems to be working and I’m now gathering Air Quality data for both inside my house and out. Unfortunately, I did not have it working in time to get readings before and after a good smoke session came through from local forest fires, but Its recording now for future events.

The last thing I’ll say, is I’m planning on adding a link to a demo unit on my website, so people can check it out without installing it (not that it’s hard to install if you have a raspberry pi).

Until next time!

HTTP Improvements

I have decided to improve my Flask app itself, in order to standardize the HTML pages. Before starting these changes, Kootnet Sensors used python functions to add HTML code around the Sensor data and put that into an HTML page, which was returned through a flask route. However, I learned a few months ago about the flask render_template() function, which simply put, works much better and is clearer to understand what it’s doing. It also makes customizing the HTML templates really easy! AKA you can move things around in the .html template files and have it look completely different yet maintain all the sensor data retrieval (Especially nice since my HTML and CSS skills are lacking).
I have already converted my home page (index.html) and my quick links page that shows all the sensor info and links. I also added a bit of mui CSS to get the main menu bar and have the mui loaded from the local sensor, so it still works when not connected online.
Long story short, this is setting an awesome foundation for my flask App.

Once I’m mostly done with the flask render templates, I’ll start looking at HTML and CSS in more depth to improve the look and feel of the HTML pages. Once that’s done, I’ll actually have a half-decent web app! In between these other improvements, I’ll slowly add more function to the pages like configuration updates, display access and even using the local database to create an HTML Plotly graph.

HTTP Authentication!

I finally got around to putting in a bit of security. Now people can’t just go around rebooting, shutting down or changing the settings on my Sensors. I have worked Basic HTTP Authorization into the programs. However, due to myself wanting to keep more information open, I have only added required authentication on things that could change what the sensors doing at the time. AKA you can get all sensor readings without logging in and even most configurations, but you can’t change any configurations or see the WiFi settings without logging in.
The Control Center program also now supports the Authentication and uses it by default in any commands sent. The user and password can be changed in both programs.
To make the program communication actually secure, I would have to enable HTTPS as well, because the password is sent in clear text over HTTP … but that’s where a problem occurs because if the DateTime is wrong on the sensor (by default they don’t keep track of time when powered off), the HTTPS won’t work… I’ll probably add it later anyway, but I’ll have to see if I can at least get the Control Center to ignore invalid certificate’s so one can at least sync the time correctly… I’ll have to think a bit more about that… Maybe I can just do 3rd party encryption on the password before sending it … yeah, I’ll have to think about that one more.

In other news, I had to make a few changes with the new Plotly 4.0 out as well as a new matplotlib module. The Plotly one was fairly easy but the matplotlib seems to freeze up my program now when doing plots (plot works fine, but can’t do anything else while it plots). I’ll have to do a bit more research and play around with it to get that proper again since it was nice being able to check other aspects and sensors during a live graph.

Since I do a lot of sensor reports, I re-worked the main screen to have a drop-down selector for the reports and database download. I don’t like the layout but I do like the convenience. I also changed the sensor commands to drop-down menu’s as well to help with the cluttered button arrangement. I’m still not sure I super like the overall look and feel … so I guess I’ll have to add GUI design to my research “ToDo” list.

I suppose my next bit of poking will be to bring some of the control center features into the web app itself, so more can be done without the control center. A few cool things to add would be a web-based SSH terminal to the local unit, the ability to edit the local configuration files and creating an offline Plotly graph right from the sensor.

Until next time!