26.09.21

  1. Docker
    1. Learn about it
    2. Do a demo
    3. Situate docker to the project at hand, and make notes, explain
      1. How to use it with influxDB
  2. InfluxDB
    1. Learn about it
    2. Do a demo
    3. Situate influxDB to the project at hand, make notes, explain
      1. How to use it with Docker
      2. How does the data ‘stream’ work in practice with my esp32
        1. What does influxDB see of the data coming from esp32
        2. What are best practices for storing data in InfluxDB
        3. What format to use?
  3. Grafana
    1. Learn about it
    2. Situate Grafana to the project at hand, make notes, explain
      1. How does it work with InfluxDB
      2. How to use it in a way that is useful for data analysis
        1. which visualizations
        2. can the content ‘created’ by grafana be exported for future use?
  4. Integration / esp32
    1. What to consider in the nitty-gritty of sending data over wlan
      1. best practices
      2. data format
      3. security
      4. power consumption
    2. plan and code new modules for data gathering and sanitization.
      1. Where to clean the gathered data? On device or in Docker? It needs to be done before InfluxDB either way.
        • is this necessary? Since the data is essentially time series real time data stream. I’m not sure where the data could be corrupted? Or maybe validation checks at esp32 and at docker if there is something weird with over air transmission

26.10.06: Update with chatGPT

I asked chatgpt about the ‘big picture’ of this project. I gave it the specs on using influx and grafana with the esp32, on how I’m planning on implementing this stack, and why. I originally started the dicussion because I was not sure about the data format. Whether to have the esp 32 send time series data (measurements), or just ‘snapshots’ of the system state over time, like overviews of how the system has been in use over the last sync period. It might be apparent by the division of the previous sentence, we settled on the latter data format. And because of that some changes needed to be made to the architecture.

  1. InfluxDB is best suited to time series data

Chatgpt Idea for the architecture

ESP32
  │
  │ Wi-Fi, when awake
  ▼
HTTP API / ingestion service
  │
  │ validate + normalize
  ▼
Database
  │
  ├── Grafana → visualization
  │
  └── SQL / other tools → analysis/manipulation
  

Inside ESP32:

ESP32 flash/NVS/SD
       │
       │ upload when Wi-Fi/server available
       ▼
   ingestion API
       │
       └── acknowledge successful upload

The ESP32 section means that the data needs to sit in a persistent outbox/queue until connection is made and the items are then popped. Heres a library for esp32 + littleFS which facilitates it: registry.platformio.org

[!NOTE] NB on the library “marcogallotta/pqueue”

  1. Not a general-purpose embedded queue. The AppendLog compaction model assumes bursty write / batch-drain cycles. Continuous random enqueue/pop will work but is not the tuned path.
  2. Not concurrent. All operations serialise. A background compaction task would still block foreground I/O – LittleFS has no parallelism. Drive compaction from your main loop instead.
  1. Which means that the items are not processed one by one but as a burst. The full outbox file is emptied at once. I assume that in the backend the context of datetime will be understood, even though the data might arrive non-chronologically.
  2. I assume this means that the outbox needs to be cleared on shutdown, so that it doesn’t interfere with the main loop.

Chatgpt idea of containers:

 Docker Compose
┌─────────────────────────────────────────────┐
│                                             │
│  ┌───────────────┐                          │
│  │ API           │                          │
│  │ FastAPI       │                          │
│  └───────┬───────┘                          │
│          │                                  │
│          ▼                                  │
│  ┌───────────────┐      ┌───────────────┐  │
│  │ PostgreSQL    │◄─────│ Grafana       │  │
│  └───────────────┘      └───────────────┘  │
│                                             │
└─────────────────────────────────────────────┘

[!NOTE] Where to run docker? Why not on my phone? I turned my old phone into a portable docker server This seems more involved, but is feasible in the future. Solves the problem of dual booting on laptop causing the docker to not be online for most of the time. Rooting old oneplus 9 is not a problem since the sim card slot doesn’t work, if that is necessary. To read: tailscape

HTTP API/FastAPI:

receive
   ↓
authenticate
   ↓
validate
   ↓
deduplicate
   ↓
insert
   ↓
acknowledge

New game plan overview

  1. Data format: JSON
  2. ESP32 sends reports on usage divided by active/deactive time.
    • one event is a timestamp and title, when something changes in the system state. ie power on or active mode
      • the wlan should be always available, so on power on the system can retrieve datetime.
    • one session is the time between power on and power off, and within that the events tell the accurate playing habits.
  3. Data is sent to HTTP API
    • the inner workings of this is still not clear to me
    • Using fastAPI?
    • handles receiving data and passing it on to database
    • could start very simple, as just passing it through, and add validation etc functionality later
  4. PostgreSQL as database
    • object relational database.
<
Previous Post
neulatuntimittari.2.01 docker
>
Next Post
sänkyrunko.01 lyhennysoperaatio