Saltar al contenido principal

HLS_MONITORING

Ver en Git

HLS Monitoring System

Overview​

This system monitors new HLS (Harmonized Landsat Sentinel) images and processes them automatically.

Architecture​

Components​

  1. SNS Webhook (features/apis/general/sns.py)

    • Receives SNS notifications from AWS
    • Auto-confirms subscriptions
    • Processes notifications and checks SQS for missed messages

  2. HLS Processor (features/services/hls_processor.py)

    • Core processing logic for HLS notifications
    • Extracts tile IDs from various data formats
    • Calls process_new_hls_image(tile_id, data) for each new image

  3. SQS Poller (features/services/sqs.py)

    • Continuously polls SQS queue for messages
    • Backup mechanism to catch any missed SNS notifications

Setup​

1. Configure SNS Topic​

Set the HLS_TOPIC_ARN in your environment:

export HLS_TOPIC_ARN="arn:aws:sns:us-west-2:673253540267:public-c2-level-3-tile-notify-v2"

2. Subscribe to SNS Topic​

Use the API endpoint:

POST /api/crop-detection/trigger/sns/force-subscription
Authorization: Bearer <your-token>

3. Run SQS Poller (Optional)​

For continuous monitoring:

python features/services/sqs.py

How It Works​

  1. New HLS image is uploaded to S3
  2. SNS notification is sent to webhook
  3. Webhook processes notification and extracts tile ID
  4. Webhook also checks SQS queue for any missed messages
  5. process_new_hls_image(tile_id, data) is called with the tile information

Processing New Images​

Edit features/services/hls_processor.py and implement your logic in:

def process_new_hls_image(tile_id, data):
    """
    Args:
        tile_id: Zone/tile identifier (e.g., T10SEG)
        data: Full notification data
    """
    # Your processing logic here
    pass

Monitoring​

  • Webhook endpoint: /sns/hls-webhook
  • List subscriptions: GET /sns/force-subscription
  • Unsubscribe: POST /sns/unsubscribe

SQS Queue​

Queue URL: https://sqs.us-east-1.amazonaws.com/162358772839/hls-notifications-queue

The webhook automatically checks this queue on each notification to ensure no messages are missed.