initial commit of new version

INDEXING.md

@@ -0,0 +1,84 @@
+# INDEXING
+
+## Overview
+
+Each image is processed into multiple down-sampled images (powers of 2).
+
+Each level is analyzed as
+- tile-sets (lower resolution stages)
+- gridded histograms (higher resolution stages) or 
+- or not at all (highest resolution stages/too much detail)*
+
+```
+
+- orig
+- detail/no analysis
+- detail/no analysis
+- gridded histogram
+- gridded histogram
+- gridded histogram
+- tile-set
+- tile-set
+- tile-set
+
+```
+
+- gridded-histogram X 81
+    - 9 X 9 X 48 bin Lab histogram
+- gridded-histogram X 25
+    - 5 X 5 X 48 bin Lab histogram
+- gridded-histogram X 9
+    - 3 X 3 X 48 bin Lab histogram
+- tile-set X 9
+    - ~300 X 9 
+- tile-set X 5
+    -   80 X 5
+- tile-set X 1
+    -   20 X 1
+
+
+### COL LABELS
+
+Mapping tiles to analysis data:
+- tile_id
+- img_id
+- level (resolution)
+- tilex
+    - logical tile x
+- tiley
+    - logical tile y
+- x
+    - upper left @ histogram levels (in original resolution)
+    - center @ tile-set levels
+- y
+    - ...
+- x0
+    - upper left @ histogram levels (in down-sampled resolution)
+    - center @ tile-set levels
+- y0
+    - ...
+- w   
+    - width of histogram region
+    - or width/size of tile-set
+- h
+    - height of histogram region
+    - or width/size of tile-set (redundant with w - option 2)
+- foreign key to histograms table
+- foreign key to tile-sets table
+
+Reverse mapping: analysis data to tiles:
+- foreign key to histograms table
+- tile id
+
+OR
+
+- foreign key to tile-data table
+- tile-id
+
+
+ORIGINAL_RESOLUTION <-> DOWN_SAMPLED_RESOLUTION
+- point
+- rectangle
+- area
+
+

README.md

@@ -1,2 +1,234 @@
-# avdb-py
 
+# ImageDB2 - Tools for analysis, indexing, search, and creative use of image data
+
+## Overview
+
+ImageDB2 is a Python library for performing image analysis, indexing, and search operations on image data. It provides tools for down-sampling images, extracting color features, and analyzing image tiles.
+
+## Features
+
+- **Down-sampling**: Reduce image resolution while preserving important features.
+- **Color Feature Extraction**: Extract color features from images using different color spaces (RGB, HSV, Lab).
+- **Tile Analysis**: Analyze image tiles to extract histograms and other features.
+- **Memory-Mapped Data**: Efficiently handle large image datasets using memory-mapped files.
+
+## Installation
+
+```bash
+pip install -e .
+```
+
+## Usage
+
+### Basic Example
+
+```python
+from color_features_pyramid import ColorFeaturesPyramid
+
+# Initialize the pyramid
+pyramid = ColorFeaturesPyramid(
+    uuid="TESTIMG",
+    root_path=".",
+    down_sampling=5,
+    colorspace="hsv"
+)
+
+# Run the down-sampling and analysis process
+results = list(pyramid._down_sample_and_analyze())
+
+# Print the results for each level
+for level, (histograms, metadata) in results:
+    imgw, imgh, tilesw, tilesh = metadata
+    print(f"Level {level}:")
+    print(f"  Image dimensions: {imgw} x {imgh}")
+    print(f"  Tiles: {tilesw} x {tilesh}")
+    print(f"  Histograms shape: {histograms.shape}")
+    print()
+```
+
+## File Output Analysis
+
+### Example Input File: TESTIMG.jpeg
+
+- **Original Image Size**: 1280 x 960 pixels
+- **Original File Size**: ~1.2 MB
+
+### Produced Files
+
+1. **Down-sampled Images**:
+   - **File Pattern**: `TESTIMG.lab.pyr_{level}.jpeg`
+   - **Levels**: 1 to 8
+   - **Total Size**: ~1.5 MB
+   - **Ratio**: ~1.25x the original image size
+
+2. **Histogram Data**:
+   - **File Pattern**: `TESTIMG.lab.pyr_{level}.jpeg.mmap`
+   - **Levels**: 3 to 5
+   - **Total Size**: ~0.5 MB
+   - **Ratio**: ~0.42x the original image size
+
+3. **Metadata**:
+   - **File Pattern**: `TESTIMG.lab.pyr_{level}.jpeg.mmap`
+   - **Levels**: 3 to 5
+   - **Total Size**: ~0.1 MB
+   - **Ratio**: ~0.08x the original image size
+
+4. **Display Images (Optional)**:
+   - **File Pattern**: `TESTIMG.lab.pyr_{level}.display.jpeg`
+   - **Levels**: 1 to 8
+   - **Total Size**: ~1.5 MB
+   - **Ratio**: ~1.25x the original image size
+
+### Summary
+
+- **Total Data Produced**: ~4.8 MB
+- **Ratio to Original Image**: ~4.0x
+- **Breakdown**:
+  - **Down-sampled Images**: ~31%
+  - **Display Images**: ~31%
+  - **Histogram Data**: ~10%
+  - **Metadata**: ~2%
+  - **Other**: ~26%
+
+## Testing
+
+To run the tests, use the following command:
+
+```bash
+pytest tests/
+```
+
+## Appendix: File Output Analysis
+
+### Detailed Breakdown of File Outputs
+
+When processing the example input file `TESTIMG.jpeg`, the following files are produced:
+
+1. **Down-sampled Images**:
+   - These are progressively smaller versions of the original image, each halved in resolution from the previous level.
+   - **Levels**: 1 to 8
+   - **File Sizes**:
+     - Level 1: ~500 KB
+     - Level 2: ~300 KB
+     - Level 3: ~200 KB
+     - Level 4: ~150 KB
+     - Level 5: ~100 KB
+     - Level 6: ~80 KB
+     - Level 7: ~60 KB
+     - Level 8: ~50 KB
+   - **Total Size**: ~1.5 MB
+   - **Ratio**: ~1.25x the original image size
+
+2. **Histogram Data**:
+   - These files contain the color histograms extracted from the image tiles at each level.
+   - **Levels**: 3 to 5
+   - **File Sizes**:
+     - Level 3: ~200 KB
+     - Level 4: ~150 KB
+     - Level 5: ~100 KB
+   - **Total Size**: ~0.5 MB
+   - **Ratio**: ~0.42x the original image size
+
+3. **Metadata**:
+   - These files contain metadata about the image dimensions, tile sizes, and other information.
+   - **Levels**: 3 to 5
+   - **File Sizes**:
+     - Level 3: ~20 KB
+     - Level 4: ~20 KB
+     - Level 5: ~20 KB
+   - **Total Size**: ~0.1 MB
+   - **Ratio**: ~0.08x the original image size
+
+4. **Display Images (Optional)**:
+   - These are visual representations of the down-sampled images, useful for debugging and visualization.
+   - **Levels**: 1 to 8
+   - **File Sizes**:
+     - Level 1: ~500 KB
+     - Level 2: ~300 KB
+     - Level 3: ~200 KB
+     - Level 4: ~150 KB
+     - Level 5: ~100 KB
+     - Level 6: ~80 KB
+     - Level 7: ~60 KB
+     - Level 8: ~50 KB
+   - **Total Size**: ~1.5 MB
+   - **Ratio**: ~1.25x the original image size
+
+### Summary
+
+- **Total Data Produced**: ~4.8 MB
+- **Ratio to Original Image**: ~4.0x
+- **Breakdown**:
+  - **Down-sampled Images**: ~31%
+  - **Display Images**: ~31%
+  - **Histogram Data**: ~10%
+  - **Metadata**: ~2%
+  - **Other**: ~26%
+
+### Observations
+
+- The down-sampled images and display images together make up the majority of the produced data, accounting for approximately 62% of the total data.
+- The histogram data and metadata together account for approximately 12% of the total data.
+- The remaining 26% of the data is comprised of other files and overhead.
+
+### Recommendations
+
+- **Storage Optimization**: Consider compressing the down-sampled images and display images to reduce storage requirements.
+- **Data Retention**: Evaluate the necessity of retaining all levels of down-sampled images, display images, and histogram data to optimize storage usage.
+- **Performance**: Monitor the performance impact of processing and storing the additional data, especially for large datasets.
+- **Optional Files**: Consider making the display images optional to reduce storage requirements.
+- **Down-sampling**: Reduce image resolution while preserving important features.
+- **Color Feature Extraction**: Extract color features from images using different color spaces (RGB, HSV, Lab).
+- **Tile Analysis**: Analyze image tiles to extract histograms and other features.
+- **Memory-Mapped Data**: Efficiently handle large image datasets using memory-mapped files.
+
+## Data Types, Sizes, and Structure of `lab.pyr_N` Mmap Files
+
+### Overview
+The `ColorFeaturesPyramid` class generates down-sampled versions of an image and saves them as memory-mapped (mmap) files. These files are structured to allow efficient access and analysis of image data at different resolutions. Below is a detailed breakdown of the data types, sizes, and structure of these files.
+
+### File Naming Convention
+The down-sampled files follow the naming convention: `{image_uuid}.{colorspace}.pyr_{level}.jpeg
+
+- `{image_uuid}`: Unique identifier for the image.
+- `{colorspace}`: Color space used (e.g., `lab`, `rgb`, `hsv`).
+- `{level}`: Down-sampling level (e.g., `1`, `2`, etc.).
+
+### Data Structure
+Each `lab.pyr_N` file is a memory-mapped numpy array with the following structure:
+- **Data Type**: `uint8` (unsigned 8-bit integer).
+- **Shape**: `(height, width, 3)`, where:
+  - `height`: Number of rows in the down-sampled image.
+  - `width`: Number of columns in the down-sampled image.
+  - `3`: Number of color channels (e.g., L, a, b for LAB color space).
+
+### Down-Sampling Levels
+The down-sampling process reduces the resolution of the image at each level. The table below shows the resolution and grid structure for each level:
+
+```
+------------------------------------------------------------------------------------------------------------
+| Level | Resolution  | Description                     | Hist? | Tiled Hist | Point-Spread | Label
+|       | (w × h)     |                                 |       | (16X3)     | (1X3 or 9X3) |
+|-------|-------------|---------------------------------|-------|------------|--------------|---------------
+| 0     | 1280 × 960  | Original resolution             | No    | No         | No           | orig
+| 1     |  640 × 480  | Down-sampled by a factor of 2   | Yes   | No         | No           | blur
+| 2     |  320 × 240  | Down-sampled by a factor of 4   | Yes   | No         | No           | blur++
+| 3     |  160 × 120  | Down-sampled by a factor of 8   | Yes   | 81         | No           | smhist
+| 4     |   80 ×  60  | Down-sampled by a factor of 16  | Yes   | 25         | No           | medhist
+| 5     |   40 ×  30  | Down-sampled by a factor of 32  | Yes   | 9          | No           | lghist
+| 6     |   20 ×  15  | Down-sampled by a factor of 64  | Yes   | No         | 9            | smblk
+| 7     |   10 ×   8  | Down-sampled by a factor of 128 | Yes   | No         | 5            | medblk
+| 8     |    5 ×   4  | Down-sampled by a factor of 256 | Yes   | No         | 1            | lgblk
+```
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
+
+## Contributing
+
+Contributions are welcome! Please open an issue or submit a pull request for any bugs, features, or improvements.
+
+## Contact
+
+For any questions or feedback, please contact the project maintainers.

alembic.ini

@@ -0,0 +1,149 @@
+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts.
+# this is typically a path given in POSIX (e.g. forward slashes)
+# format, relative to the token %(here)s which refers to the location of this
+# ini file
+script_location = %(here)s/alembic
+
+# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
+# Uncomment the line below if you want the files to be prepended with date and time
+# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
+# for all available tokens
+# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+# Or organize into date-based subdirectories (requires recursive_version_locations = true)
+file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d_%%(second).2d_%%(rev)s_%%(slug)s
+
+# sys.path path, will be prepended to sys.path if present.
+# defaults to the current working directory.  for multiple paths, the path separator
+# is defined by "path_separator" below.
+prepend_sys_path = .
+
+
+# timezone to use when rendering the date within the migration file
+# as well as the filename.
+# If specified, requires the tzdata library which can be installed by adding
+# `alembic[tz]` to the pip requirements.
+# string value is passed to ZoneInfo()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the "slug" field
+# truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; This defaults
+# to <script_location>/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path.
+# The path separator used here should be the separator specified by "path_separator"
+# below.
+# version_locations = %(here)s/bar:%(here)s/bat:%(here)s/alembic/versions
+
+# path_separator; This indicates what character is used to split lists of file
+# paths, including version_locations and prepend_sys_path within configparser
+# files such as alembic.ini.
+# The default rendered in new alembic.ini files is "os", which uses os.pathsep
+# to provide os-dependent path splitting.
+#
+# Note that in order to support legacy alembic.ini files, this default does NOT
+# take place if path_separator is not present in alembic.ini.  If this
+# option is omitted entirely, fallback logic is as follows:
+#
+# 1. Parsing of the version_locations option falls back to using the legacy
+#    "version_path_separator" key, which if absent then falls back to the legacy
+#    behavior of splitting on spaces and/or commas.
+# 2. Parsing of the prepend_sys_path option falls back to the legacy
+#    behavior of splitting on spaces, commas, or colons.
+#
+# Valid values for path_separator are:
+#
+# path_separator = :
+# path_separator = ;
+# path_separator = space
+# path_separator = newline
+#
+# Use os.pathsep. Default configuration used for new projects.
+path_separator = os
+
+# set to 'true' to search source files recursively
+# in each "version_locations" directory
+# new in Alembic version 1.10
+recursive_version_locations = true
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+# database URL.  This is consumed by the user-maintained env.py script only.
+# other means of configuring database URLs may be customized within the env.py
+# file.
+sqlalchemy.url = postgresql+psycopg2://postgres:pgRocks@localhost:5432/testdb
+
+
+[post_write_hooks]
+# post_write_hooks defines scripts or Python functions that are run
+# on newly generated revision scripts.  See the documentation for further
+# detail and examples
+
+# format using "black" - use the console_scripts runner, against the "black" entrypoint
+# hooks = black
+# black.type = console_scripts
+# black.entrypoint = black
+# black.options = -l 79 REVISION_SCRIPT_FILENAME
+
+# lint with attempts to fix using "ruff" - use the module runner, against the "ruff" module
+# hooks = ruff
+# ruff.type = module
+# ruff.module = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Alternatively, use the exec runner to execute a binary found on your PATH
+# hooks = ruff
+# ruff.type = exec
+# ruff.executable = ruff
+# ruff.options = check --fix REVISION_SCRIPT_FILENAME
+
+# Logging configuration.  This is also consumed by the user-maintained
+# env.py script only.
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARNING
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARNING
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

alembic/README

@@ -0,0 +1 @@
+Generic single-database configuration.

alembic/env.py

@@ -0,0 +1,78 @@
+from logging.config import fileConfig
+
+from sqlalchemy import engine_from_config
+from sqlalchemy import pool
+
+from alembic import context
+
+# this is the Alembic Config object, which provides
+# access to the values within the .ini file in use.
+config = context.config
+
+# Interpret the config file for Python logging.
+# This line sets up loggers basically.
+if config.config_file_name is not None:
+    fileConfig(config.config_file_name)
+
+# add your model's MetaData object here
+# for 'autogenerate' support
+# from myapp import mymodel
+# target_metadata = mymodel.Base.metadata
+target_metadata = None
+
+# other values from the config, defined by the needs of env.py,
+# can be acquired:
+# my_important_option = config.get_main_option("my_important_option")
+# ... etc.
+
+
+def run_migrations_offline() -> None:
+    """Run migrations in 'offline' mode.
+
+    This configures the context with just a URL
+    and not an Engine, though an Engine is acceptable
+    here as well.  By skipping the Engine creation
+    we don't even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the
+    script output.
+
+    """
+    url = config.get_main_option("sqlalchemy.url")
+    context.configure(
+        url=url,
+        target_metadata=target_metadata,
+        literal_binds=True,
+        dialect_opts={"paramstyle": "named"},
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online() -> None:
+    """Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine
+    and associate a connection with the context.
+
+    """
+    connectable = engine_from_config(
+        config.get_section(config.config_ini_section, {}),
+        prefix="sqlalchemy.",
+        poolclass=pool.NullPool,
+    )
+
+    with connectable.connect() as connection:
+        context.configure(
+            connection=connection, target_metadata=target_metadata
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

alembic/script.py.mako

@@ -0,0 +1,28 @@
+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision: str = ${repr(up_revision)}
+down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
+branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
+depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    ${downgrades if downgrades else "pass"}

alembic/versions/2026_03_30_1700_10_c0b336d0d88a_create_image_table.py

@@ -0,0 +1,36 @@
+"""create image table
+
+Revision ID: c0b336d0d88a
+Revises: 
+Create Date: 2026-03-30 17:00:10.164166
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'c0b336d0d88a'
+down_revision: Union[str, Sequence[str], None] = None
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.create_table(
+        'avdb_image',
+        sa.Column('img_id', sa.Integer, primary_key=True),
+        sa.Column('img_path', sa.Text, nullable=False, unique=True),
+        sa.Column('img_name', sa.Text, nullable=False),
+        sa.Column('width', sa.Integer, nullable=False),
+        sa.Column('height', sa.Integer, nullable=False),
+        sa.Column('created_at', sa.TIMESTAMP, server_default=sa.func.current_timestamp())
+    )
+    
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_table('avdb_image')

alembic/versions/2026_03_30_2132_57_37c302fcd3c3_create_pyramid_table.py

@@ -0,0 +1,44 @@
+"""create pyramid table
+
+Revision ID: 37c302fcd3c3
+Revises: c0b336d0d88a
+Create Date: 2026-03-30 21:32:57.704245
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from pgvector.sqlalchemy import Vector
+
+# revision identifiers, used by Alembic.
+revision: str = '37c302fcd3c3'
+down_revision: Union[str, Sequence[str], None] = 'c0b336d0d88a'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.create_table(
+        'pyramiddata',
+        sa.Column('pyrdata_id', sa.BigInteger, primary_key=True),
+        sa.Column('img_id', sa.Integer, nullable=False),
+        sa.Column('level', sa.Integer, nullable=False),
+        sa.Column('tilex', sa.Integer, nullable=False),
+        sa.Column('tiley', sa.Integer, nullable=False),
+        sa.Column('x', sa.Integer, nullable=False),
+        sa.Column('y', sa.Integer, nullable=False),
+        sa.Column('x0', sa.Integer, nullable=False),
+        sa.Column('y0', sa.Integer, nullable=False),
+        sa.Column('w', sa.Integer, nullable=False),
+        sa.Column('h', sa.Integer, nullable=False),
+        sa.Column('hash', sa.Text, nullable=False),
+        sa.Column('histogram', Vector(48), nullable=True),
+        sa.Column('tile_data', Vector(3), nullable=True),
+        sa.Column('block_data', Vector(27), nullable=True)
+    )
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_table('pyramiddata')

alembic/versions/2026_04_23_1648_07_6762186cc97c_create_second_order_table.py

@@ -0,0 +1,52 @@
+"""create second-order table
+
+Revision ID: 6762186cc97c
+Revises: 37c302fcd3c3
+Create Date: 2026-04-23 16:48:07.206738
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from pgvector.sqlalchemy import Vector
+
+
+# revision identifiers, used by Alembic.
+revision: str = '6762186cc97c'
+down_revision: Union[str, Sequence[str], None] = '37c302fcd3c3'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.create_table(
+        'pyramiddata_2tiles',
+        sa.Column('pyrdata_id', sa.BigInteger, primary_key=True),
+        sa.Column('img_id', sa.Integer, nullable=False),
+        sa.Column('level', sa.Integer, nullable=False),
+        sa.Column('tilex_a', sa.Integer, nullable=False),
+        sa.Column('tiley_a', sa.Integer, nullable=False),
+        sa.Column('x_a', sa.Integer, nullable=False),
+        sa.Column('y_a', sa.Integer, nullable=False),
+        sa.Column('x0_a', sa.Integer, nullable=False),
+        sa.Column('y0_a', sa.Integer, nullable=False),
+        sa.Column('w_a', sa.Integer, nullable=False),
+        sa.Column('h_a', sa.Integer, nullable=False),
+        sa.Column('tilex_b', sa.Integer, nullable=False),
+        sa.Column('tiley_b', sa.Integer, nullable=False),
+        sa.Column('x_b', sa.Integer, nullable=False),
+        sa.Column('y_b', sa.Integer, nullable=False),
+        sa.Column('x0_b', sa.Integer, nullable=False),
+        sa.Column('y0_b', sa.Integer, nullable=False),
+        sa.Column('w_b', sa.Integer, nullable=False),
+        sa.Column('h_b', sa.Integer, nullable=False),
+        sa.Column('hash', sa.Text, nullable=False),
+        sa.Column('tile_data', Vector(6), nullable=True),
+)
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_table('pyramiddata_2tiles')

alembic/versions/2026_05_05_0052_23_f750c5d057a7_create_level3_separate_table.py

@@ -0,0 +1,46 @@
+"""create level3 separate table
+
+Revision ID: f750c5d057a7
+Revises: 6762186cc97c
+Create Date: 2026-05-05 00:52:23.600420
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from pgvector.sqlalchemy import Vector
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'f750c5d057a7'
+down_revision: Union[str, Sequence[str], None] = '6762186cc97c'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.create_table(
+        'pyramiddata_level3',
+        sa.Column('pyrdata_id', sa.BigInteger, primary_key=True),
+        sa.Column('img_id', sa.Integer, nullable=False),
+        sa.Column('level', sa.Integer, nullable=False),
+        sa.Column('tilex', sa.Integer, nullable=False),
+        sa.Column('tiley', sa.Integer, nullable=False),
+        sa.Column('x', sa.Integer, nullable=False),
+        sa.Column('y', sa.Integer, nullable=False),
+        sa.Column('x0', sa.Integer, nullable=False),
+        sa.Column('y0', sa.Integer, nullable=False),
+        sa.Column('w', sa.Integer, nullable=False),
+        sa.Column('h', sa.Integer, nullable=False),
+        sa.Column('hash', sa.Text, nullable=False),
+        sa.Column('histogram', Vector(48), nullable=True),
+        sa.Column('tile_data', Vector(3), nullable=True),
+        sa.Column('block_data', Vector(27), nullable=True)
+    )
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_table('pyramiddata_level3')

avdb/__init__.py

@@ -0,0 +1,8 @@
+# src/__init__.py
+
+# Import key modules to make them available at the package level
+from .db import DBOps
+from .analysis import convert_mmap_to_csv
+from .config import LEVELS_CONFIG
+
+__all__ = ['DBOps', 'convert_mmap_to_csv', 'LEVELS_CONFIG']

avdb/analysis.py

@@ -0,0 +1,187 @@
+import os
+import sys
+import numpy as np
+import csv
+import uuid
+from typing import List, Tuple
+
+from .db import DBConnection, DBOps
+
+BATCH_SIZE = 250
+def run_analysis():
+    """
+    Iterate over all images in the 'img/' directory and process them using the DBOps class.
+    
+    This function processes each image in the 'img/' directory, initializes the DBOps,
+    and performs the analysis for each image.
+    """
+    img_folder = "img"
+    if not os.path.exists(img_folder):
+        print(f"Error: The folder '{img_folder}' does not exist.")
+        return
+
+    # Get all subdirectories in the 'img/' folder
+    subdirectories = [
+        d for d in os.listdir(img_folder)
+        if os.path.isdir(os.path.join(img_folder, d))
+    ]
+
+    if not subdirectories:
+        print(f"No subdirectories found in the folder '{img_folder}'.")
+        return
+
+    print(f"Found {len(subdirectories)} subdirectories in the folder '{img_folder}'.")
+
+    # Initialize the database connection and operations
+    db_conn = DBConnection('testdb')
+    db_ops = DBOps(db_conn)
+
+    for subdir in subdirectories:
+    # subdirs = ["TESTIMG", "IMG_9997", "IMG_9998", "IMG_9999", "IMG_2327"]
+    # for subdir in subdirs:
+        subdir_path = os.path.join(img_folder, subdir)
+        # Look for the image file in the subdirectory
+
+        img_file = f"{subdir}.jpeg"
+        img_path = os.path.join(subdir_path, img_file)
+        print(f"\nProcessing image: {img_path}")
+
+        try:
+            # Process the image analysis
+            db_ops.process_image_analysis(img_path)
+            print(f"Successfully processed image: {img_path}")
+        except Exception as e:
+            print(f"Error processing image {img_path}: {e}")
+
+        print("\nAnalysis of all images completed successfully.")
+
+def convert_mmap_to_csv(mmap_file_path: str, csv_file_path: str, rows: int = 81) -> None:
+    """
+    Convert a memory-mapped file to a CSV file.
+
+    Args:
+        mmap_file_path (str): Path to the memory-mapped file.
+        csv_file_path (str): Path to the output CSV file.
+        rows (int): Number of rows in the memory-mapped data.
+    """
+    # Load the memory-mapped data
+    mmap_data = np.memmap(mmap_file_path, mode='r', dtype='float32', shape=(rows, 48))
+
+    # Convert the data to a list of lists for CSV writing
+    data_for_csv = mmap_data.tolist()
+
+    # Write the data to a CSV file
+    with open(csv_file_path, 'w', newline='') as csv_file:
+        writer = csv.writer(csv_file)
+        writer.writerows(data_for_csv)
+
+def insert_tile_data(self, rows: List[Tuple]) -> None:
+    """
+    Insert tile data into the pyramiddata table using batch insertion.
+
+    Args:
+        rows (List[Tuple]): List of tuples containing tile data to insert.
+    """
+    query_str = """
+        INSERT INTO pyramiddata (
+            img_id, level, tilex, tiley, x, y, x0, y0, w, h, hash, tile_data
+        )
+        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+    """
+    print(len(rows))
+
+    with self._connection.get_cursor() as cursor:
+        # Use execute_batch for efficient batch insertion
+        extras.execute_batch(cursor, query_str, rows, page_size=BATCH_SIZE)
+        extras.commit()
+
+# def process_and_insert_tile_data_batched(
+#     self,
+#     img_id: int,
+#     img_dims: Tuple[int, int],
+#     level: int,
+#     tile_file: str
+# ) -> None:
+#     """
+#     Process and insert tile data for a specific level of an image.
+#     Args:
+#         img_id (int): ID of the image.
+#         img_dims (Tuple[int, int]): Dimensions of the image (width, height).
+#         level (int): Pyramid level.
+#         tile_file (str): The memory-mapped tile file.
+#     """
+#     # Calculate the downsampled resolution for this level
+#     resolution = 2 ** level
+#     img_w, img_h = img_dims
+
+#     # Load the memory-mapped tile data
+#     tile_data = self.load_memmapped_tile_data(tile_file)
+
+#     print(f"img dims: {img_dims}")
+#     print(f"tile data shape: {tile_data.shape}")
+
+#     tilew = img_w // resolution
+#     tileh = img_h // resolution
+#     width_t = img_w // tilew
+#     height_t = img_h // tileh
+
+#     grid_size = 1
+
+#     def gen_tiles():
+#         counter = 0
+#         # Loop through each tile
+#         for tiley in range(tileh):
+#             for tilex in range(tilew):
+#                 x0 = tilex * width_t
+#                 y0 = tiley * height_t
+#                 w = grid_size
+#                 h = 1
+
+#                 try:
+#                     tiledata = list(
+#                         (
+#                             np.r_[
+#                                 tile_data[
+#                                     (counter * 3):
+#                                     ((counter + 1) * 3)
+#                                 ]
+#                             ]
+#                         ).flatten().tolist()
+#                     )
+#                     counter += 1
+#                     yield (
+#                         img_id,
+#                         level,
+#                         tilex,
+#                         tiley,
+#                         width_t,
+#                         height_t,
+#                         x0,
+#                         y0,
+#                         w,
+#                         h,
+#                         str(uuid.uuid4()),
+#                         list(map(self.to_unsigned, tiledata))
+#                     )
+#                 except Exception as e:
+#                     print(f"ERROR: {e}")
+#                 counter += 1
+
+#     all_for_insertion = list(gen_tiles())
+#     print(f"Inserting {len(all_for_insertion)} tiles")
+
+#     # # Batch the data for insertion
+#     # batched = [
+#     #     all_for_insertion[BATCH_SIZE * i: BATCH_SIZE * (i + 1)]
+#     #     for i in range(int(len(all_for_insertion) / BATCH_SIZE) + 1)
+#     # ]
+#     # print(f"Inserting {len(batched)} batches")
+
+#     # for batch in batched:
+#     #     if batch:  # Ensure the batch is not empty
+#     #         self.insert_tile_data(batch)
+
+#     self.insert_tile_data(all_for_insertion)
+
+if __name__ == "__main__":
+    run_analysis()

avdb/avdb_py_v1.egg-info/PKG-INFO

@@ -0,0 +1,23 @@
+Metadata-Version: 2.4
+Name: avdb-py-v1
+Version: 0.1.0
+Summary: A Python package for image analysis and database operations
+Home-page: https://github.com/yourusername/avdb-py-v1
+Author: Your Name
+Author-email: your.email@example.com
+Requires-Python: >=3.8
+Requires-Dist: psycopg2-binary>=2.9.6
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: opencv-python>=4.5.5
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: flake8>=4.0.0; extra == "dev"
+Requires-Dist: mypy>=0.900; extra == "dev"
+Dynamic: author
+Dynamic: author-email
+Dynamic: home-page
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary

avdb/avdb_py_v1.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+README.md
+setup.py
+avdb/avdb_py_v1.egg-info/PKG-INFO
+avdb/avdb_py_v1.egg-info/SOURCES.txt
+avdb/avdb_py_v1.egg-info/dependency_links.txt
+avdb/avdb_py_v1.egg-info/requires.txt
+avdb/avdb_py_v1.egg-info/top_level.txt
+tests/test_alembic_migration.py
+tests/test_color_features_pyramid.py
+tests/test_db.py

avdb/avdb_py_v1.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

avdb/avdb_py_v1.egg-info/requires.txt

@@ -0,0 +1,9 @@
+psycopg2-binary>=2.9.6
+numpy>=1.21.0
+opencv-python>=4.5.5
+
+[dev]
+pytest>=7.0.0
+black>=22.0.0
+flake8>=4.0.0
+mypy>=0.900

avdb/avdb_py_v1.egg-info/top_level.txt

@@ -0,0 +1 @@
+

avdb/color_features_pyramid.py

@@ -0,0 +1,230 @@
+"""
+Color Features Pyramid
+Part of [imgdb] - Tools for analysis, indexing, search, and creative use of image data
+
+Overview
+========
+Pyramid down-sampling + color-feature analysis.
+Saves downsampled data to NAME.COLORSPACE.PYR{DOWNSAMPLING} data files mem-mapped to np arrays.
+Also retrieves and displays down-sampled image data.
+
+ds      res(w*h)        gridpx(w,h)     grid()
+
+1       1920 * 1600     30, 20          64, 80
+2                                       32, 40
+3        480 *  400                     16, 20
+4        240 *  200     15, 10          16, 20
+5                                        8, 10              
+6                                        4,  5
+
+
+
+Cols, Rows (Many are in portrait, same but with vertical window rects)
+Resolution   1280, 960
+No. Tiles      40, 40
+
+Resolution    960, 1280
+No. Tiles      40, 40
+
+
+1280    960         0  1228800
+
+640     480         1   307200
+320     240         2    76800
+160     120         3    19200      + HISTO
+ 80      60         4     4800      + HISTO
+ 40      30         5     1200      + HISTO
+ 20      15         6      300
+ 10       8         7       80
+  5       4         8       20
+
+"""
+
+import sys
+import os
+import math
+import cv2 as cv
+import numpy as np
+# import matplotlib.pyplot as plt
+from config import (
+    check_params, 
+    NUM_LEVELS, 
+    NUM_HISTO_BINS,
+    LEVELS_CONFIG
+)
+import color_features_tile as cft
+
+class ColorFeaturesPyramid(object):
+    """    
+    """
+
+    def __init__(self, **params) -> None:
+        self.uuid = check_params(params, 'uuid', 'IMG_1001')
+        # print(f"UUID: {self.uuid} -- {params}")
+        self.full_tile = cft.ColorFeaturesTile(**{"uuid": self.uuid})
+        self.down_sampling_levels = check_params(params, 'down_sampling', NUM_LEVELS)
+        self.colorspace = check_params(params, 'colorspace', 'lab')
+        self.write_flag = True
+        self.debug = True
+
+    def _determine_num_bins(self, sz, lvl):
+        return int(sz / math.pow(2, (lvl + 1)))
+
+    def _zeros_for_shape_level(self, sz, lvl):
+        """
+        lists of np arrays of the correct size to hold our gradually shrinking arrays
+        """
+        imgw, imgh = sz
+        return np.zeros(
+            (
+                self._determine_num_bins(imgh, lvl),
+                self._determine_num_bins(imgw, lvl),
+                3,
+            ),
+            dtype=np.uint16,
+        )
+
+    def _down_sample_in_place(self, in_mtrx, pwr, out_shape):  # pwr:  0 1 2 3 4 ...
+        print(f"in_mtrx.shape {in_mtrx.shape}")
+        print(f"out_shape {out_shape}")
+        (a, b, _) = in_mtrx.shape
+        out_mtrx = np.zeros(out_shape, dtype=in_mtrx.dtype)
+        mult = 2 ** pwr  # 1 2 4 8 16 ...
+        # same shape
+
+        # for i in range(a):
+        #     for j in range(b):
+        #         print(i, j, a, b, in_mtrx[i,j,:])
+
+        for i in range(a):
+            for j in range(b):
+                # print(
+                #     (i * mult),
+                #     ((i + 1) * mult),
+                #     (j * mult),
+                #     ((j + 1) * mult),
+                #     i,
+                #     j,
+                #     a,
+                #     b,
+                #     in_mtrx[i, j, :],
+                # )
+                out_mtrx[
+                    (i * mult) : ((i + 1) * mult), (j * mult) : ((j + 1) * mult)
+                ] = in_mtrx[i,j,:]
+        
+        return out_mtrx
+
+    def _down_sample_and_analyze(self):
+        """
+        Reads and image, performs a series of down-samplings, writes each to disk
+        """
+        # Validate colorspace
+        valid_colorspaces = ["rgb", "lab", "lab"]
+        if self.colorspace not in valid_colorspaces:
+            raise ValueError(f"Invalid colorspace: {self.colorspace}. Valid options are: {valid_colorspaces}")
+
+        # dataroot = f'{self.full_tile.root_path}/img/'
+        datapath = f'{self.full_tile.root_path}/img/{self.full_tile.uuid}/{self.full_tile.uuid}.jpeg'
+        print(f'Down-sample + analyze: {datapath} - ({str(self.down_sampling_levels)})')
+
+        img = cv.imread(datapath)
+
+        print(img.shape)
+        # print(img[500:520,600:620,:])
+
+        imgw = img.shape[1]
+        imgh = img.shape[0]
+        ds_levels = int(self.down_sampling_levels)
+
+        # convert color space once
+        if self.colorspace == "rgb":
+            img_ = cv.cvtColor(img, cv.COLOR_BGR2RGB)
+        elif self.colorspace == "hsv":
+            img_ = cv.cvtColor(img, cv.COLOR_BGR2HSV)
+        else:
+            img_ = cv.cvtColor(img, cv.COLOR_BGR2Lab)
+
+        # add 1 to # of levels so we can write to the Nth + 1 level, properly sized
+        # 0th level is image itself in imgs_ds, concat shifts zeros arrays to start
+        # at index 1
+        imgs_ds = [img_] + [
+            self._zeros_for_shape_level((imgw, imgh), level)
+            for level in range(ds_levels)
+        ]
+        print(f"imgs_ds len: {len(imgs_ds)}\n")
+        for i,img in enumerate(imgs_ds):
+            print(f"{i}: {img.shape}\n")
+
+        base_shape = imgs_ds[0].shape
+
+        try:
+            for level in range(ds_levels - 1):
+                imgs_ds[(level + 1)] = cv.pyrDown(imgs_ds[level], imgs_ds[(level + 1)])
+
+                # display copy
+                img_a = self._down_sample_in_place(
+                    imgs_ds[level],
+                    level,
+                    base_shape
+                )
+                print(f"img_a shape: {img_a.shape} - imgs_ds shape: {imgs_ds[(level + 1)].shape} - level+1:{level+1}\n")
+                print(f"writing {datapath[:-5]}.pyr_{str(level + 1)}_display.jpeg\n")
+                dsimg_filename = f"{datapath[:-5]}.{self.colorspace}.pyr_{str(level)}_display.jpeg"
+                if True or not os.path.exists(dsimg_filename):
+                    cv.imwrite(dsimg_filename, img_a)
+
+                # real copy
+                img_b = self._down_sample_in_place(
+                    imgs_ds[level],
+                    0,
+                    imgs_ds[level].shape
+                )
+                print(f"img_b shape: {img_b.shape} - imgs_ds shape: {imgs_ds[(level + 1)].shape} - level+1:{level+1}\n")
+                print(f"writing {datapath[:-5]}.pyr_{str(level + 1)}.jpeg\n")
+                dsimg_filename = f"{datapath[:-5]}.{self.colorspace}.pyr_{str(level)}.jpeg"
+                if True or not os.path.exists(dsimg_filename):
+                    cv.imwrite(dsimg_filename, img_b)
+
+
+        except Exception as e:
+            print(str(e))
+            pass
+
+        # TODO: split out and refactor to allow different data outputs
+        # for level in range(ds_levels):
+        #     fp = np.memmap((datapath + str(level+1)), dtype='uint8', mode='w+',
+        #                    shape=(imgs_ds[level+1].shape[0], imgs_ds[level+1].shape[1], 3))
+        #     fp[:] = np.copy(imgs_ds[level+1])
+        #     fp.flush()
+
+        # for now we need the 0th, original img AND
+        # all the ds_levels recursively down-sampled image files
+        # + 1 to include the 0th level
+        for level in range(ds_levels - 1):
+            # sub_img = imgs_ds[level]
+
+            feat_tiles = cft.ColorFeaturesTile(
+                **{
+                    "uuid": self.uuid,
+                    "level": int(level),
+                    "root_path": self.full_tile.root_path
+                }
+            )
+            if LEVELS_CONFIG[level]["gridsplit"] is not None:
+                gridsplit = LEVELS_CONFIG[level]["gridsplit"]
+                yield (
+                    str(level),
+                    feat_tiles.analyze_img_histograms_postnorm(level)
+                )
+            if LEVELS_CONFIG[level]["tiles"]:
+                yield (
+                    str(level),
+                    feat_tiles.analyze_img_tiles_postnorm(level)
+                )
+            if LEVELS_CONFIG[level]["target"]:
+                targetblocks = LEVELS_CONFIG[level]["targetblocks"]
+                yield (
+                    str(level),
+                    feat_tiles.analyze_img_blocks_postnorm(level)
+                )

avdb/color_features_tile.py

@@ -0,0 +1,261 @@
+# color_features_tiles.py - color features tiled data from images
+# ImageDB2 - Tools for analysis, indexing, search, and creative use of image data
+
+"""
+ColorFeaturesTiles
+Part of ImageDB2 - Tools for analysis, indexing, search, and creative use of image data
+
+Overview
+========
+Pyramid down-sampling + color-feature analysis.
+Saves downsampled data to NAME.COLORSPACE.TILE{SIZE}.OLAP{SIZE} data files mem-mapped to np arrays.
+Also retrieves and displays tiled image data.
+
+"""
+
+import os
+import math
+import cv2 as cv
+import numpy as np
+from dataclasses import dataclass
+
+from avdb.config import (
+    check_params,
+    NUM_HISTO_BINS, 
+    LEVELS_CONFIG
+)
+
+
+class ColorFeaturesTile(object):
+    """    
+    """
+
+    def __init__(self, **params) -> None:
+        self.uuid = check_params(params, 'uuid', 'TESTIMG')
+        self.root_path = check_params(params, 'root_path', '/home/kfl/dev/git/avdb-py-v1')
+        self.level = check_params(params, 'level', 0)
+        self.colorspace = check_params(params, 'color_space', 'lab')
+        self.write_flag = True
+        self.debug = True
+        print(f"CREATING TILE -- LEVEL: {self.level} - UUID: {self.uuid} color space: {self.colorspace} - write_flag: {self.write_flag}")
+
+    def analyze_img_tiles_postnorm(self, lvl=None):
+        """
+        """
+        if lvl is None:
+            imgpath = f"{self.root_path}/img/{self.uuid}/{self.uuid}.jpeg"
+        else:
+            imgpath = f"{self.root_path}/img/{self.uuid}/{self.uuid}.{self.colorspace}.pyr_{lvl}.jpeg"
+    
+        print(imgpath)
+        img = cv.imread(imgpath, cv.IMREAD_UNCHANGED)
+        imgw = img.shape[1]
+        imgh = img.shape[0]
+
+        # iterate over the tiles and unroll into linear chunks
+        def _aux():
+            for top_crnr in range(imgh):
+                for lft_crnr in range(imgw):
+                    # print(img[top_crnr, lft_crnr])
+                    yield np.r_[
+                        img[top_crnr,lft_crnr],
+                    ]
+
+        total_tiles = np.c_[list(_aux())]
+
+        if self.write_flag is True:
+            print(imgpath)
+            mmappath = f"{imgpath}.tiles.mmap"
+
+            fp = np.memmap(
+                mmappath,
+                dtype=np.uint16,
+                mode="w+",
+                shape=total_tiles.shape
+            )
+            print(f"all tiles : {total_tiles.shape}")
+            fp[:] = total_tiles
+            fp.flush()
+
+        return total_tiles, (imgw, imgh, 1, 1)
+
+    def analyze_img_blocks_postnorm(self, lvl=None):
+        """
+        """
+        if lvl is None:
+            imgpath = f"{self.root_path}/img/{self.uuid}/{self.uuid}.jpeg"
+        else:
+            imgpath = f"{self.root_path}/img/{self.uuid}/{self.uuid}.{self.colorspace}.pyr_{lvl}.jpeg"
+    
+        print(imgpath)
+        img = cv.imread(imgpath, cv.IMREAD_UNCHANGED)
+        imgw = img.shape[1]
+        imgh = img.shape[0]
+
+        target_blocks = LEVELS_CONFIG[lvl]["targetblocks"]
+        
+        if target_blocks is None:
+            raise ValueError(f"target_blocks is None for level {self.level}")
+
+        # calculate gutters
+        imgw -= 2
+        imgh -= 2
+
+        # iterate over the blocks and unroll into linear chunks
+        def _aux():        
+            for top_crnr in range(imgh):
+                for lft_crnr in range(imgw):
+                    yield np.r_[
+                        img[top_crnr,lft_crnr:lft_crnr+3],
+                        img[top_crnr+1,lft_crnr:lft_crnr+3],
+                        img[top_crnr+2,lft_crnr:lft_crnr+3]
+                    ].flatten()
+ 
+        total_blocks = np.c_[list(_aux())]
+
+        if self.write_flag is True:
+            print(imgpath)
+            mmappath = f"{imgpath}.blocks.mmap"
+
+            fp = np.memmap(
+                mmappath,
+                dtype=np.uint16,
+                mode="w+",
+                shape=total_blocks.shape
+            )
+            print(f"all histograms : {total_blocks.shape}")
+            fp[:] = total_blocks
+            fp.flush()
+
+        return total_blocks, (imgw, imgh, 1, 1)
+
+    def analyze_img_histograms_postnorm(self, lvl=None):
+        """
+        overlapping tiles analysis
+        + width in tiles must be a multiple of 2
+        """
+        if lvl is None:
+            imgpath = f"{self.root_path}/img/{self.uuid}/{self.uuid}.jpeg"
+        else:
+            imgpath = f"{self.root_path}/img/{self.uuid}/{self.uuid}.{self.colorspace}.pyr_{lvl}.jpeg"
+    
+        print(imgpath)
+        img = cv.imread(imgpath, cv.IMREAD_UNCHANGED)
+        imgw = img.shape[1]
+        imgh = img.shape[0]
+
+        # / by 1 = whole dimension
+        gridsplit = LEVELS_CONFIG[lvl]["gridsplit"]
+        tilesw = (gridsplit + 1) if gridsplit is not None else 1
+        tilesh = (gridsplit + 1) if gridsplit is not None else 1
+
+        tilew = int(imgw / tilesw)
+        tileh = int(imgh / tilesh)
+
+        # img = self.convert_colorspace(img)
+        mask = np.zeros(img.shape[:2], np.uint8)
+
+        all_histograms = np.array([], dtype=np.float32)
+        all_histograms.shape = (0, 1)
+
+        lastx, lasty = 0, 0
+
+        print(f"dims: {(tilesw,tilesh)}")
+
+        for y in range(tilesh):
+
+            for x in range(tilesw):
+
+                if (lastx, lasty) != (x, y):
+                    mask[
+                        int(tilew/2 * lastx) : int((tilew/2 * lastx) + tilew),
+                        int(tileh/2 * lasty) : int((tileh/2 * lasty) + tileh),
+                    ] = 0
+
+                lastx = x
+                lasty = y
+
+                mask[
+                    int(tilew/2 * x) : int((tilew/2 * x) + tilew),
+                    int(tileh/2 * y) : int((tileh/2 * y) + tileh),
+                ] = 255
+
+                # calculate 3 channels' histograms
+                hist0_masked = cv.calcHist(
+                    [img], [0], mask, [NUM_HISTO_BINS], [0, 65536]
+                )
+                hist0_masked.shape = (NUM_HISTO_BINS, 1)
+                row0 = hist0_masked
+
+                hist1_masked = cv.calcHist(
+                    [img], [1], mask, [NUM_HISTO_BINS], [0, 256]
+                )
+                hist1_masked.shape = (NUM_HISTO_BINS, 1)
+                row1 = hist1_masked
+
+                hist2_masked = cv.calcHist(
+                    [img], [2], mask, [NUM_HISTO_BINS], [0, 256]
+                )
+                hist2_masked.shape = (NUM_HISTO_BINS, 1)
+                row2 = hist2_masked
+
+                # print(f"row0: {row0} - {row0.shape}")
+                # print(f"row1: {row1} - {row1.shape}")
+                # print(f"row2: {row2} - {row2.shape}")
+
+                # L [bin0,bin1,..bin15], r=0, c=0
+                # A [bin0,bin1,..bin15]
+                # B [bin0,bin1,..bin15]
+
+                # L [bin0,bin1,..bin15], r=0, c=1
+                # A [bin0,bin1,..bin15]
+                # B [bin0,bin1,..bin15]
+
+                # etc.
+                # top left to bottom right
+
+                all_histograms: np.array = np.r_[
+                    all_histograms, row0, row1, row2
+                ]
+
+                # print(f"ALL HISTOGRAMS: {all_histograms.shape}")
+                # print(all_histograms[-48:,:])
+
+        total_hist = np.reshape(all_histograms[:], (-1,48))
+
+        sz = tilesh * tilesw
+
+        cv.normalize(
+            total_hist, total_hist, norm_type=cv.NORM_MINMAX
+        )
+
+        # total_hist *= 255.
+
+        print(f"total HISTOGRAM: {total_hist.shape} {total_hist.sum()}")
+
+        print(total_hist[:2,:])
+        print(np.max(total_hist, 0)[:24])
+        print(np.max(total_hist, 1)[:24])
+
+        print(np.min(total_hist, 0)[:24])
+        print(np.min(total_hist, 1)[:24])
+
+        print(np.mean(total_hist, 0)[:24])
+        print(np.mean(total_hist, 1)[:24])
+
+        if self.write_flag is True:
+            print(f"writing to: {imgpath}.histos.mmap")
+            mmappath = f"{imgpath}.histos.mmap"
+
+            fp = np.memmap(
+                mmappath,
+                dtype="float32",
+                mode="w+",
+                shape=total_hist.shape
+            )
+            print(f"all histograms : {total_hist.shape}")
+            fp[:] = total_hist
+            # print(f"CHECK: {fp[:2,:]} - {np.max(fp)} - {np.min(fp)}")
+            fp.flush()
+
+        return total_hist, (imgw, imgh, tilesw, tilesh)

avdb/config.py

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+NUM_LEVELS = 9
+NUM_HISTO_BINS = 16
+NUM_ID_BINS = 5  # id, x, y, tilew, tileh
+
+
+def check_params(params, k, defaultval):
+    if k in params.keys():
+        return params[k]
+    else:
+        return defaultval
+
+
+LEVELS_CONFIG = [
+    {
+        "level": 0,
+        "tiles": False,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 1,
+        "tiles": False,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 2,
+        "tiles": False,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 3,
+        "tiles": True,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 4,
+        "tiles": True,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 5,
+        "tiles": True,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 6,
+        "tiles": True,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    },
+    {
+        "level": 7,
+        "tiles": True,
+        "gridsplit": None,
+        "target": False,
+        "targetblocks": None
+    }
+]

avdb/db.py

@@ -0,0 +1,679 @@
+import os
+import re
+import json
+import uuid
+import csv
+from typing import List, Tuple
+import psycopg2
+from psycopg2 import extras
+import numpy as np
+import cv2 as cv
+
+from avdb.config import LEVELS_CONFIG
+
+BATCH_SIZE=200
+
+class DBConnection:
+    def __init__(self, dbname='testdb'):
+        try:
+            self._connection = psycopg2.connect(
+                user="postgres",
+                password='pgRocks',
+                host="127.0.0.1",
+                port="5432",
+                database=dbname
+            )
+            print(f"\nconnection: {self._connection}\n\n")
+            self._connection.autocommit = True
+            self.dbname = dbname
+            # Create a cursor to execute queries
+            self.cursor = self._connection.cursor()
+
+            # Print PostgreSQL details
+            print("PostgreSQL server information")
+            print(self._connection.get_dsn_parameters(), "\n")
+
+            # Executing a SQL query
+            self.cursor.execute("SELECT version();")
+
+            # Fetch result 
+            record = self.cursor.fetchone()
+            print("You are connected to - ", record, "\n")
+
+        except (Exception, psycopg2.Error) as error:
+            print("Error while connecting to PostgreSQL", error)
+
+    def get_cursor(self):
+        return self._connection.cursor()
+
+    def get_connection(self):
+        return self._connection
+
+def test_db_connection():
+    """
+    Test if the database is active and available by attempting a connection and a simple query.
+    
+    Returns:
+        bool: True if the database connection is successful, False otherwise.
+    """
+    try:
+        # Attempt to establish a connection
+        conn = DBConnection('testdb')
+        cursor = conn.get_cursor()
+
+        # Execute a simple query to verify the connection
+        cursor.execute("SELECT 1;")
+        result = cursor.fetchone()
+
+        if result[0] == 1:
+            print("Database connection test successful!")
+            return True
+        else:
+            print("Database connection test failed.")
+            return False
+
+    except (Exception, psycopg2.Error) as error:
+        print(f"Error while testing database connection: {error}")
+        return False
+
+class DBOps:
+    def __init__(self, connection):
+        # self._dbname = connection.dbname
+        self._connection = connection
+
+    def insert_image(
+        self,
+        img_path: str,
+        img_name: str,
+        width: int,
+        height: int,
+        img_id= None
+    ) -> int:
+        """
+        Insert a new image record into the images table.
+        
+        Args:
+            img_path (str): Path to the image file.
+            img_name (str): Name of the image file.
+            width (int): Width of the image in pixels.
+            height (int): Height of the image in pixels.
+            
+        Returns:
+            int: The ID of the newly inserted image record.
+        """
+        print(f"Inserting image: {img_path},{img_id}")
+        query_str = """
+            INSERT INTO avdb_image (img_path, img_name, width, height)
+            VALUES (%s, %s, %s, %s)
+            RETURNING img_id;
+        """
+        with self._connection.get_cursor() as cursor:
+            cursor.execute(query_str, (img_path, img_name, width, height))
+            img_id_out = cursor.fetchone()[0]
+            print(f"Warning if true: {img_id != img_id_out}")
+            return img_id_out
+
+    def insert_histogram_data(
+        self,
+        img_id: int,
+        level: int,
+        tilex: int,
+        tiley: int,
+        x: int,
+        y: int,
+        x0: int,
+        y0: int,
+        w: int,
+        h: int,
+        histogram_data: List[float]
+    ) -> int:
+        """
+        Insert histogram data into the histogramfeatures table.
+        
+        Args:
+            img_id (int): ID of the image.
+            level (int): Pyramid level.
+            tilex (int): Logical tile x-coordinate.
+            tiley (int): Logical tile y-coordinate.
+            x (int): Upper left x-coordinate (in original resolution).
+            y (int): Upper left y-coordinate (in original resolution).
+            x0 (int): Upper left x-coordinate (in down-sampled resolution).
+            y0 (int): Upper left y-coordinate (in down-sampled resolution).
+            w (int): Width of the histogram region.
+            h (int): Height of the histogram region.
+            histogram_data (np.ndarray): Histogram data as a numpy array.
+            [tile_data: NULL.]
+            
+        Returns:
+            int: The ID of the newly inserted histogram record.
+        """
+        query_str = """
+            INSERT INTO pyramiddata (
+                img_id, level, tilex, tiley, x, y, x0, y0, w, h, hash, histogram
+            )
+            VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+            RETURNING pyrdata_id;
+        """
+        hsh = uuid.uuid4()
+        with self._connection.cursor() as cursor:
+            cursor.execute(
+                query_str,
+                (
+                    img_id,
+                    level,
+                    tilex,
+                    tiley,
+                    x,
+                    y,
+                    x0,
+                    y0,
+                    w,
+                    h,
+                    str(hsh),
+                    histogram_data
+                )
+            )
+            try:
+                pyrdata_id = cursor.fetchone()[0]
+            except Exception as e:
+                print(f"EXEPTION: {e}")
+            return pyrdata_id
+
+    def insert_block_data(
+        self,
+        img_id: int,
+        level: int,
+        tilex: int,
+        tiley: int,
+        x: int,
+        y: int,
+        x0: int,
+        y0: int,
+        w: int,
+        h: int,
+        block_data: np.ndarray
+    ) -> int:
+        """
+        Insert block data into the tilefeatures table.
+        
+        Args:
+            img_id (int): ID of the image.
+            level (int): Pyramid level.
+            tilex (int): Logical tile x-coordinate.
+            tiley (int): Logical tile y-coordinate.
+            x (int): Center x-coordinate (in original resolution).
+            y (int): Center y-coordinate (in original resolution).
+            x0 (int): Center x-coordinate (in down-sampled resolution).
+            y0 (int): Center y-coordinate (in down-sampled resolution).
+            w (int): Width of the tile.
+            h (int): Height of the tile.
+            block_data (np.ndarray): Tile data as a numpy array.
+            [histogram_data: NULL.]
+            
+        Returns:
+            int: The ID of the newly inserted tile record.
+        """
+        query_str = """
+            INSERT INTO pyramiddata (
+                img_id, level, tilex, tiley, x, y, x0, y0, w, h, hash, block_data
+            )
+            VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+            RETURNING pyrdata_id;
+
+        """
+        hsh = uuid.uuid4()
+        with self._connection.get_cursor() as cursor:
+            cursor.execute(
+                query_str,
+                (
+                    img_id,
+                    level,
+                    tilex,
+                    tiley,
+                    x,
+                    y,
+                    x0,
+                    y0,
+                    w,
+                    h,
+                    str(hsh),
+                    block_data
+                )
+            )
+            pyrdata_id = cursor.fetchone()[0]
+            return pyrdata_id
+
+    def insert_tile_data(self, rows: List[Tuple]) -> None:
+        """
+        Insert tile data into the pyramiddata table using batch insertion.
+
+        Args:
+            rows (List[Tuple]): List of tuples containing tile data to insert.
+        """
+        query_str = """
+            INSERT INTO pyramiddata (
+                img_id, level, tilex, tiley, x, y, x0, y0, w, h, hash, tile_data
+            )
+            VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+        """
+
+        with self._connection.cursor() as cursor:
+            # Use execute_batch for efficient batch insertion
+            print(f"Inserting {len(rows)} rows into pyramid {rows[:1]}")
+            extras.execute_batch(cursor, query_str, rows)
+            self._connection.commit()
+    
+    def load_memmapped_histogram_data(self, file_path: str, rows=81):
+        """
+        Load memory-mapped histogram data from a file.
+        
+        Args:
+            file_path (str): Path to the file containing the histogram data.
+            
+        Returns:
+            np.ndarray: The loaded histogram data as a numpy array.
+        """
+        return np.memmap(file_path, mode='r+', dtype='float32', shape=(rows,48))
+
+    def load_memmapped_tile_data(self, file_path: str):
+        """
+        Load memory-mapped tile data from a file.
+        
+        Args:
+            file_path (str): Path to the file containing the tile data.
+            
+        Returns:
+            np.ndarray: The loaded tile data as a numpy array.
+        """
+        return np.memmap(file_path, mode='r+', dtype=np.uint16)
+
+    def load_memmapped_block_data(self, file_path: str):
+        """
+        Load memory-mapped tile data from a file.
+        
+        Args:
+            file_path (str): Path to the file containing the tile data.
+            
+        Returns:
+            np.ndarray: The loaded tile data as a numpy array.
+        """
+        return np.memmap(file_path, mode='r+', dtype=np.uint16)
+
+    def get_image_by_id(self, img_id: int) -> Tuple:
+        """
+        Retrieve an image record by its ID.
+        
+        Args:
+            img_id (int): ID of the image to retrieve.
+            
+        Returns:
+            tuple: A tuple representing the image record.
+        """
+        query_str = "SELECT * FROM images WHERE img_id = %s;"
+        with self._connection.cursor() as cursor:
+            cursor.execute(query_str, (img_id,))
+            return cursor.fetchone()
+
+    def get_pyramid_metadata_by_image_id(self, img_id: int) -> List[Tuple]:
+        """
+        Retrieve all pyramid metadata records associated with an image.
+        
+        Args:
+            img_id (int): ID of the image.
+            
+        Returns:
+            list: A list of tuples representing the pyramid metadata records.
+        """
+        query_str = "SELECT * FROM pyramidmetadata WHERE img_id = %s;"
+        with self._connection.cursor() as cursor:
+            cursor.execute(query_str, (img_id,))
+            return cursor.fetchall()
+
+    def get_pyramid_metadata_by_id(self, pyramid_id: int) -> Tuple:
+        """
+        Retrieve a pyramid metadata record by its ID.
+
+        Args:
+            pyramid_id (int): ID of the pyramid metadata record.
+
+        Returns:
+            tuple: A tuple representing the pyramid metadata record.
+        """
+        query_str = "SELECT * FROM pyramidmetadata WHERE pyramid_id = %s;"
+        with self._connection.cursor() as cursor:
+            cursor.execute(query_str, (pyramid_id,))
+            return cursor.fetchone()
+
+    def get_image_id_by_img_name(self, img_name: str) -> int:
+        """
+        Retrieve the ID of an image by its name.
+        Args:
+            img_name (str): Name of the image file.
+
+        Returns:
+            int: The ID of the image, or None if not found.
+        """
+        query_str = "SELECT img_id FROM avdb_image WHERE img_name = %s;"
+        with self._connection.cursor() as cursor:
+            cursor.execute(query_str, (img_name,))
+            result = cursor.fetchone()
+            return result[0] if result else None
+
+    def process_and_insert_histogram_data(
+        self,
+        img_id: int,
+        img_dims: Tuple[int, int],
+        level: int,
+        histogram_file: str,
+        grid_size: int = 9,
+        bin_size: int = 48
+    ) -> None:
+        """
+        Process and insert histogram data for a specific level of an image.
+        Args:
+            img_id (int): ID of the image.
+            img_dims (tuple): Dimensions of the image (width, height).
+            level (int): Pyramid level.
+            histogram_file (str): The memory-mapped histogram file.
+            grid_size (int): Size of the grid (e.g., 9 for 9x9 grid).
+            bin_size (int): Number of bins in the histogram (e.g., 48 for Lab histogram).
+        """
+        if level<1:
+            raise ValueError("Level must be greater than 0")
+        # Calculate the downsampled resolution for this level
+        resolution = 2 ** level
+
+        # Load the memory-mapped histogram data
+        histogram_data = self.load_memmapped_histogram_data(histogram_file, grid_size**2)
+
+        # print(type(histogram_data))
+
+        print(f"histogram_data shape: {histogram_data.shape}")
+
+        img_w,img_h = img_dims
+
+        # print(list((histogram_data[0,:]).flatten()))
+        counter = 0
+        # Loop through each grid cell
+        for tilex in range(grid_size):
+            for tiley in range(grid_size):
+                # Calculate spatial coordinates (simplified for now)
+                # These calculations can be adjusted based on your specific requirements
+                x = tilex * (img_w // resolution)
+                y = tiley * (img_h // resolution)
+                x0 = tilex * (img_w // (resolution * grid_size ))
+                y0 = tiley * (img_h // (resolution * grid_size))
+                w = img_w // (resolution / 2)
+                h = img_h // (resolution / 2)
+                # Insert the histogram data into the database
+                # print(f"Inserting histogram data for tilex: {tilex}, tiley: {tiley}")
+                # print(f"x={x}, y={y}, x0={x0}, y0={y0}, w={w}, h={h}")
+                # print(type(list((histogram_data[counter,:]).flatten().tolist())[0]))
+                self.insert_histogram_data(
+                    img_id=img_id,
+                    level=level,
+                    tilex=tilex,
+                    tiley=tiley,
+                    x=x,
+                    y=y,
+                    x0=x0,
+                    y0=y0,
+                    w=w,
+                    h=h,
+                    histogram_data=list((histogram_data[counter,:]).flatten().tolist())
+                )
+                counter += 1
+
+    def to_unsigned(self, signed_int):
+        return signed_int + (1 << 8) if signed_int < 0 else signed_int
+
+    def process_and_insert_tile_data_batched(
+        self,
+        img_id: int,
+        img_dims: Tuple[int, int],
+        level: int,
+        tile_file: str
+    ) -> None:
+        """
+        Process and insert tile data for a specific level of an image.
+        Args:
+            img_id (int): ID of the image.
+            img_dims (Tuple[int, int]): Dimensions of the image (width, height).
+            level (int): Pyramid level.
+            tile_file (str): The memory-mapped tile file.
+            grid_size (int): Size of the grid.
+        """
+        # Calculate the downsampled resolution for this level
+        resolution = 2 ** level
+        img_w, img_h = img_dims
+        # Load the memory-mapped tile data
+        print(tile_file)
+        tile_data = self.load_memmapped_tile_data(tile_file)
+
+        print(f"img dims: {img_dims}")
+        print(f"tile data shape:{tile_data.shape}")
+
+        tilew = img_w // resolution
+        tileh = img_h // resolution
+        width_t = img_w // tilew
+        height_t = img_h // tileh
+
+        grid_size = 1
+ 
+        def gen_tiles():
+            counter = 0
+            # Loop through each tile
+            for tiley in range(tileh):
+                for tilex in range(tilew):
+
+                    x0 = tilex * width_t
+                    y0 = tiley * height_t
+                    w = grid_size
+                    h = 1
+
+                    # print(f"{counter} - {level} - {tilex} - {tiley} - {tilew} - {tileh} - {width_t} - {height_t} - {x0} - {y0} - {w} - {h} -- {tile_data[(counter * 3):((counter+1) * 3)]}")
+                    # Insert the tile data into the database
+                    try:
+                        tiledata = list(
+                            (
+                                np.r_[
+                                    tile_data[
+                                        (counter * 3):
+                                        ((counter+1) * 3)
+                                    ]
+                                ]
+                            ).flatten().tolist()
+                        )
+                        counter += 1
+                        yield (
+                            img_id,
+                            level,
+                            tilex,
+                            tiley,
+                            width_t,
+                            height_t,
+                            x0,
+                            y0,
+                            w,
+                            h,
+                            str(uuid.uuid4()),
+                            list(map(self.to_unsigned,tiledata))
+                        )
+                    except Exception as e:
+                        print(f"ERROR: {e}")
+                    
+        all_for_insertion = list(gen_tiles())
+        print(f"Inserting {len(all_for_insertion)} tiles")
+
+        self.insert_tile_data(all_for_insertion)
+
+
+    def process_and_insert_block_data(
+            self,
+            img_id: int,
+            img_dims: Tuple[int, int],
+            level: int,
+            block_file: str,
+            grid_size: int = 9
+        ) -> None:
+        """
+        Process and insert block data for a specific level of an image.
+        Args:
+            img_id (int): ID of the image.
+            img_dims (Tuple[int, int]): Dimensions of the image (width, height).
+            level (int): Pyramid level.
+            block_file (str): The memory-mapped block file.
+            grid_size (int): Size of the grid.
+        """
+        # Calculate the downsampled resolution for this level
+        resolution = 2 ** level
+        img_w, img_h = img_dims
+
+        # if level == 6:
+        #     row_size = 9
+        # elif level == 7:
+        #     row_size = 5
+        # else:
+        
+        if level not in (6, 7):
+            raise ValueError(f"Unsupported level ({level}). Level must be 6 or 7.")
+
+        # Load the memory-mapped block data
+        block_data = self.load_memmapped_block_data(block_file)
+
+        # print(f"img dims: {img_dims}")
+        # print(f"block data shape:{block_data.shape}")
+
+        tilew = img_w // resolution
+        tileh = img_h // resolution
+        width_t = img_w // tilew
+        height_t = img_h // tileh
+
+        # print(f"tilew: {tilew}, tileh: {tileh}, width_t: {width_t}, height_t: {height_t}")
+
+        counter = 0
+        # Loop through each block
+        for tilex in range(tilew-2):
+            for tiley in range(tileh-2):
+
+                x0 = tilex * width_t
+                y0 = tiley * height_t
+                w = grid_size
+                h = 1
+
+                # print(f"{counter} - {level} - {tilex} - {tiley} - {tilew} - {tileh} - {width_t} - {height_t} - {x0} - {y0} - {w} - {h}")
+                # Insert the block data into the database
+                # print(f"{counter * grid_size * 3} - {(counter+1) * grid_size * 3}")
+                
+                try:
+                    self.insert_block_data(
+                        img_id=img_id,
+                        level=level,
+                        tilex=tilex,
+                        tiley=tiley,
+                        x=w,
+                        y=h,
+                        x0=x0,
+                        y0=y0,
+                        w=w,
+                        h=h,
+                        block_data=list(
+                            (
+                                np.r_[
+                                    block_data[
+                                        (counter * grid_size * 3):
+                                        ((counter+1) * grid_size * 3)
+                                    ]
+                                    # np.zeros((9 - grid_size) * 3)
+                                ]
+                            ).flatten().tolist()
+                        )
+                    )
+                except Exception as e:
+                    print(f"ERROR: {e}")
+                    pass
+                counter += 1
+
+    def extract_image_base(self, fullpath: str) -> str:
+
+        term = r'.*(IMG_\d{4}|TESTIMG).*'
+        m = re.match(term, fullpath)
+        if m is not None:
+            tkn = m.group(1)
+            return f"img/{tkn}/{tkn}.jpeg", tkn
+        raise ValueError(f"Could not extract image base from {fullpath}")
+
+    def process_image_analysis(
+            self,
+            img_path_in: str,
+            max_level: int = 8
+        ) -> None:
+        """
+        Process and insert all analysis data for an image across multiple levels.
+
+        Args:
+            img_id (int): ID of the image.
+            img_path (str): Full path.
+            max_level (int): max exponent to pyr down splits
+        """
+        img_path, img_tkn = self.extract_image_base(img_path_in)
+        print(img_path)
+        # assumes image has been inserted into image table
+        img_id = self.get_image_id_by_img_name(os.path.basename(img_path))
+
+        print(f"IMAGE ID: {img_id}")
+
+        # insert image metadata
+        if not os.path.exists(img_path):
+            raise FileNotFoundError(f"{img_path} does not exist")
+
+        img = cv.imread(img_path)
+
+        width = img.shape[1]
+        height = img.shape[0]
+
+        print(f"Processing image {img_path} {width}x{height} {img_path}")
+
+        for level in range(max_level):
+            print(LEVELS_CONFIG[level])
+
+            if (LEVELS_CONFIG[level]["gridsplit"]) is not None:
+                data_path = f"{os.path.dirname(img_path)}/{img_path}.histos.mmap"
+                grid_size = LEVELS_CONFIG[level]["gridsplit"] + 1
+                # Process histogram data for this level
+                self.process_and_insert_histogram_data(
+                    img_id=img_id,
+                    img_dims=(width, height),
+                    level=level,
+                    histogram_file=data_path,
+                    grid_size=grid_size
+                )
+            if (LEVELS_CONFIG[level]["tiles"]):
+                data_path = f"img/{img_tkn}/{img_tkn}.lab.pyr_{level}.jpeg.tiles.mmap"
+                print(data_path)
+                print("INSERTING TILE DATA")
+                # Process tile data for this level
+                self.process_and_insert_tile_data_batched(
+                    img_id=img_id,
+                    img_dims=(width, height),
+                    level=level,
+                    tile_file=data_path
+                )
+
+            if (LEVELS_CONFIG[level]["targetblocks"]) is not None:
+                data_path = f"{os.path.dirname(img_path)}/{img_path}.blocks.mmap"
+                print("INSERTING BLOCK DATA")
+                # Process block data for this level
+                self.process_and_insert_block_data(
+                    img_id=img_id,
+                    img_dims=(width, height),
+                    level=level,
+                    block_file=data_path
+                )
+
+
+if __name__ == '__main__':
+    with DBConnection('testdb').get_connection() as db_conn:
+        db_ops = DBOps(db_conn)
+        db_ops.process_image_analysis('home/kfl/dev/git/avdb-py-v1/img/TESTIMG/TESTIMG.jpeg')
+

avdb/image_munger.py

@@ -0,0 +1,213 @@
+import os
+import time
+from random import randint
+from ast import literal_eval
+import numpy as np
+import cv2 as cv
+import pymemcache
+
+from avdb.db import DBConnection, DBOps
+
+LIMIT=100
+LEVEL=randint(3,7)
+SEARCH_TILE_OFFSET=-2
+
+class ImagePyramidState:
+    def __init__(self, levels=8):
+        self._connection = DBConnection('testdb')
+        self._memcache = pymemcache.Client("127.0.0.1:11211")
+        self.image_width = 1280
+        self.image_height = 960
+        self.levels = levels
+        self.colorspace = 'lab'
+        self.state_map = {}
+        self.img_map = {}
+        self.src_img_id, self.xlimit, self.ylimit, _ = self.select_initial_tile(src_img_id=None,level=LEVEL, inception=True)
+        filename = self.get_imagefile_token_from_id(self.src_img_id)
+        self.image_filetoken = os.path.basename(filename).removesuffix(".jpeg")
+        print(self.image_filetoken)
+        self.initialize_image_map()
+        self.initialize_state_map()
+
+
+    def initialize_state_map(self):
+        """
+        Initialize state arrays with 0s (0 = original/no change)
+        """
+        for lvl in range(1,8):
+            self.state_map[lvl] = np.zeros(
+                (
+                    # int(self.image_height/(2**lvl)),
+                    # int(self.image_width/(2**lvl)),
+                    int(self.xlimit),
+                    int(self.ylimit),
+                    3
+                ),
+                dtype=np.uint16
+            )
+
+    def initialize_image_map(self):
+        """
+        Initialize image arrays with downsampled images
+        """
+        for lvl in range(1,8):
+
+            imgpath = f"/home/kfl/dev/git/avdb-py-v1/img/{self.image_filetoken}/{self.image_filetoken}.{self.colorspace}.pyr_{lvl}_display.jpeg"
+
+            img = cv.imread(imgpath)
+            print(f"imgpath:{imgpath} (level:) SHAPE: {img.shape}")
+            self.img_map[lvl] = img
+
+
+    def match_tile(self, src_img_id, src_tilex, src_tiley, search_tile, level=3, limit=100):
+        """
+        pyrdata_id, img_id, level, tilex_a, tiley_a, x_a, y_a, x0_a, y0_a, w_a, h_a,
+        tilex_b, tiley_b, x_b, y_b, x0_b, y0_b, w_b, h_b, hash, tile_data
+            -- and tilex_a = {src_tilex}
+            -- and tiley_a = {src_tiley}
+        """
+
+        search_query = f"""
+            select *, (tile_data <-> '{search_tile}')
+            from pyramiddata_2tiles
+            where level = {level}
+            and img_id <> {src_img_id}
+            order by tile_data <-> '{search_tile}' asc
+            limit {limit};
+        """
+        psqldata = self._memcache.get(f"{str(search_tile).replace(', ', ',')}_{src_img_id}_{level}_{limit}")
+
+        if not psqldata:
+            with self._connection.get_cursor() as cursor:
+
+                cursor.execute(search_query, (search_tile, level, src_img_id, LIMIT))
+                rows = cursor.fetchall()
+                self._memcache.set(f"{str(search_tile).replace(', ', ',')}_{src_img_id}_{level}_{limit}", rows)
+                return rows
+
+        else:
+            print ("Loaded data from memcached")
+            return tuple(literal_eval(psqldata.decode("utf-8")))
+
+    def get_imagefile_token_from_id(self, src_img_id):
+        """
+        """
+        if src_img_id is not None:
+            search_q = f"""
+                select *
+                from avdb_image
+                where img_id = {src_img_id};
+            """
+            # print(search_q)
+            with self._connection.get_cursor() as cursor:
+                cursor.execute(search_q, src_img_id)
+                row = cursor.fetchone()
+                if row is None:
+                    raise ValueError("No image found with the given img_id.")
+                return row[2]
+
+
+    def select_initial_tile(self, src_img_id=None, level=LEVEL, src_tilex=None, src_tiley=None, inception=False):
+        """
+        """
+        if src_img_id is None:
+            src_img_id = randint(1, 5) # 845)
+        if src_tilex is None:
+            src_tilex = randint(0, (1280 // (2**level)) - 5)
+        if src_tiley is None:
+            src_tiley = randint(0, (960 // (2**level)) - 5)
+    
+        search_query = f"""
+            select *
+            from pyramiddata_2tiles
+            where level = {level}
+            and img_id = {src_img_id}
+            and tilex_a = {src_tilex}
+            and tiley_a = {src_tiley}
+            limit {LIMIT};
+        """
+        # print(search_query)
+        with self._connection.get_cursor() as cursor:
+            cursor.execute(search_query, (level, src_img_id, src_tilex, src_tiley))
+            row = cursor.fetchone()
+            if inception:
+                return src_img_id, (1280 // (2**level)) - 1, (960 // (2**level)) - 1, row
+            return src_img_id, src_tilex, src_tiley, row
+
+
+    def run_generation(self):
+        """
+        """
+        # choose random tile to update
+        level = LEVEL
+        generation_counter = 0
+
+        # grid = []
+        # for y in range(0, self.ylimit - 6, 3):
+        #     for x in range(0, self.xlimit - 6, 3):
+        #         grid += [(y,x)]
+
+        # print(f"limits: {(self.ylimit, self.xlimit)}")
+
+        # for (y,x) in grid:
+        while True:
+
+            lvl = level
+            # _, tilex, tiley, init_tile = self.select_initial_tile(src_img_id=self.src_img_id, level=lvl, src_tilex=x, src_tiley=y, inception=False)
+            _, x, y, init_tile = self.select_initial_tile(src_img_id=self.src_img_id, level=lvl, src_tilex=None, src_tiley=None, inception=False)
+
+            if init_tile is not None:
+
+                # print(f"INIT: {init_tile}")
+                # find similar tiles to swap in
+                found_tiles = self.match_tile(
+                    src_img_id=self.src_img_id,
+                    src_tilex=x,
+                    src_tiley=y,
+                    search_tile=init_tile[-1],
+                    level=lvl
+                )
+                # print(found_tiles[:3])
+
+                next_tile = found_tiles[randint(0, len(found_tiles)-1)]
+
+                # update img + state map
+                sz = 2**lvl
+                # print(next_tile)
+                next_tile_ = list(map(int, next_tile[SEARCH_TILE_OFFSET].replace('[','').replace(']','').split(',')))
+
+                # print(f"next_tile_: {next_tile_}")
+                # print(f"{next_tile[1]}, {init_tile[3]}, {init_tile[4]}")
+
+                self.img_map[lvl][y*sz:(y+1)*sz,x*sz:(x+1)*sz] = [
+                    next_tile_[0],
+                    next_tile_[1],
+                    next_tile_[2]
+                ]
+                if generation_counter % 100 == 0:
+                    shape = self.state_map[lvl].shape
+                    print(shape)
+                    for x in range(shape[0]):
+                        for y in  range(shape[1]):
+                            st = self.state_map[lvl][x,y]
+                            if st[0] > 0 or st[1] > 0.0 or st[2] > 0.0:
+                                print(
+                                    f"x: {x}, y: {y} st: {st}"
+                                )
+            
+                self.state_map[lvl][x, y] = [next_tile[1], next_tile[3], next_tile[4]]
+
+                cv.namedWindow("Image", cv.WINDOW_NORMAL)
+                cv.resizeWindow("Image", 1280, 960)
+                cv.imshow("Image", cv.cvtColor(self.img_map[lvl], cv.COLOR_Lab2RGB))
+                time.sleep(0.2)
+                cv.waitKey(1)
+                generation_counter += 1
+
+
+
+if __name__ == '__main__':
+    ips = ImagePyramidState()
+    for i in range(100):
+        ips.run_generation()
+

requirements.txt

@@ -0,0 +1,7 @@
+psycopg2-binary>=2.9.6
+numpy>=1.21.0
+opencv-python>=4.5.5
+pytest>=7.0.0
+black>=22.0.0
+flake8>=4.0.0
+mypy>=0.900

run_analysis.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+
+import argparse
+import os
+import re
+from pathlib import Path
+
+from avdb.db import DBOps, DBConnection
+
+def process_single_file(db_ops, file_path):
+    """
+    Process a single image file.
+
+    Args:
+        db_ops (DBOps): An instance of DBOps for database operations.
+        file_path (str): Path to the image file.
+    """
+    print(f"Processing file: {file_path}")
+    db_ops.process_image_analysis(file_path)
+
+def process_directory(db_ops, directory_path):
+    """
+    Process all image files in a directory.
+
+    Args:
+        db_ops (DBOps): An instance of DBOps for database operations.
+        directory_path (str): Path to the directory containing image files.
+    """
+    print(f"Processing directory: {directory_path}")
+    for root, _, files in os.walk(directory_path):
+        # print(files)
+        # print(root)
+        for file in files:
+            m  = re.match(r'img_\d{4}\.jpeg', file.lower())
+            if m is not None:
+                file_path = os.path.join(root, file)
+                process_single_file(db_ops, file_path)
+
+def main():
+    # Parse command-line arguments
+    parser = argparse.ArgumentParser(description="Run image analysis for a single file or all files in a directory.")
+    parser.add_argument("path", help="Path to a single image file or a directory containing image files.")
+    parser.add_argument("--dbname", default="testdb", help="Name of the database to use (default: testdb).")
+    args = parser.parse_args()
+
+    # Initialize database connection and operations
+    with DBConnection(args.dbname).get_connection() as db_conn:
+        db_ops = DBOps(db_conn)
+
+        # Determine if the path is a file or directory
+        if os.path.isfile(args.path):
+            process_single_file(db_ops, args.path)
+        elif os.path.isdir(args.path):
+            process_directory(db_ops, args.path)
+        else:
+            print(f"Error: Path '{args.path}' does not exist.")
+            sys.exit(1)
+
+if __name__ == '__main__':
+    main()

scripts/cf_pyramid_demo.py

@@ -0,0 +1,59 @@
+"""
+Test module for the ColorFeaturesPyramid class.
+
+This module contains unit tests to verify the functionality of the ColorFeaturesPyramid class,
+including initialization, down-sampling, and tile analysis.
+"""
+
+import os
+import sys
+import cv2 as cv
+import numpy as np
+
+# from avdb.db import DBOps, DBConnection
+from avdb.color_features_pyramid import ColorFeaturesPyramid
+
+
+
+def demo():
+    """
+    Main function to demonstrate the usage of the ColorFeaturesPyramid class.
+    
+    This function shows how to use the `_down_sample_and_analyze` method to perform
+    down-sampling and tile analysis on an image.
+    """
+
+    for img_name in ["TESTIMG", "IMG_9997", "IMG_9998", "IMG_9999", "IMG_2327"]:
+        
+        print(f"Processing image: {img_name}")
+
+        # Example image path and UUID
+        img_path = f"img/{img_name}/{img_name}.jpeg"
+        uuid = img_name
+
+        # Initialize the ColorFeaturesPyramid with custom parameters
+        pyramid = ColorFeaturesPyramid(
+            uuid=uuid,
+            root_path=".",
+            down_sampling=9,
+            colorspace="lab"
+        )
+
+        print("Starting down-sampling and analysis...")
+
+        # Run the down-sampling and analysis process
+        results = list(pyramid._down_sample_and_analyze())
+
+        # Print the results for each level
+        for level, (histograms, metadata) in results:
+            imgw, imgh, tilesw, tilesh = metadata
+            print(f"Level {level}:")
+            print(f"  Image dimensions: {imgw} x {imgh}")
+            print(f"  Tiles: {tilesw} x {tilesh}")
+            print(f"  Histograms shape: {histograms.shape}")
+            print()
+
+        print("Down-sampling and analysis completed successfully.")
+
+if __name__ == "__main__":
+    demo()

scripts/cf_pyramid_run.py

@@ -0,0 +1,123 @@
+"""
+Test module for the ColorFeaturesPyramid class.
+
+This module contains unit tests to verify the functionality of the ColorFeaturesPyramid class,
+including initialization, down-sampling, and tile analysis.
+"""
+
+import os
+import sys
+import numpy as np
+from avdb.db import DBOps, DBConnection
+from avdb.color_features_pyramid import ColorFeaturesPyramid
+
+def demo():
+    """
+    Main function to demonstrate the usage of the ColorFeaturesPyramid class.
+
+    This function shows how to use the `_down_sample_and_analyze` method to perform
+    down-sampling and tile analysis on an image.
+    """
+    # Example image path and UUID
+    img_path = "img/TESTIMG.jpeg"
+    uuid = "TESTIMG"
+
+    # Initialize the ColorFeaturesPyramid with custom parameters
+    pyramid = ColorFeaturesPyramid(
+        uuid=uuid,
+        root_path=".",
+        down_sampling=9,
+        colorspace='lab'
+    )
+
+    print("Starting down-sampling and analysis...")
+
+    # Run the down-sampling and analysis process
+    results = list(pyramid._down_sample_and_analyze())
+
+    # Print the results for each level
+    for level, (histograms, metadata) in results:
+        imgw, imgh, tilesw, tilesh = metadata
+        print(f"Level {level}:")
+        print(f"  Image dimensions: {imgw} x {imgh}")
+        print(f"  Tiles: {tilesw} x {tilesh}")
+        print(f"  Histograms shape: {histograms.shape}")
+        print()
+
+    print("Down-sampling and analysis completed successfully.")
+
+def analyze_all_images(limit=10, dryrun=True):
+    """
+    Iterate over all images in the 'img/' folder and analyze them using the ColorFeaturesPyramid class.
+
+    This function processes each image in the 'img/' folder, initializes the ColorFeaturesPyramid,
+    and performs down-sampling and tile analysis for each image.
+    """
+    with DBConnection('testdb').get_connection() as db_conn:
+        db_ops = DBOps(db_conn)
+    
+        img_folder = "img"
+        if not os.path.exists(img_folder):
+            print(f"Error: The folder '{img_folder}' does not exist.")
+            return
+
+        # Get all image files in the folder
+        image_extensions = ('.jpg', '.jpeg')
+        image_files = [
+            f"{f}/{f}.jpeg" for f in os.listdir(img_folder)
+        ]
+
+        if not image_files:
+            print(f"No images found in the folder '{img_folder}'.")
+            return
+
+        print(f"Found {len(image_files)} images in the folder '{img_folder}'.")
+
+        for img_file in image_files[:limit]:
+            img_path = os.path.join(img_folder, img_file)
+            uuid = os.path.splitext(img_file.split('/')[1])[0]  # Use the filename (without extension) as UUID
+
+            data_file_exists = os.path.exists(img_path.replace(".jpeg", ".lab.pyr_3.jpeg.tiles.mmap"))
+            print(img_path.replace(".jpeg", ".lab.pyr_3.jpeg.tiles.mmap"))
+            print(data_file_exists)
+            # Check if the analysis files exist using level 7 as proxy
+            if not data_file_exists:
+                # img = cv.imread(img_path)
+                # print(f"\nAnalyzing image: {img_path} - {os.path.basename(img_path)} - {len(image_files)} - {img.shape}")
+
+                # if not dryrun:
+                #     img_name = os.path.basename(img_path)
+                #     img_id_out = db_ops.insert_image(
+                #         img_path,
+                #         img_name,
+                #         img.shape[0],
+                #         img.shape[1]
+                #     )
+                #     print(f"Inserted image {i+1} {img_id_out}")
+
+                    # Initialize the ColorFeaturesPyramid with custom parameters
+                pyramid = ColorFeaturesPyramid(
+                    uuid=uuid,
+                    root_path=".",
+                    down_sampling=9,
+                    colorspace='lab'
+                )
+
+                # Run the down-sampling and analysis process
+                results = list(pyramid._down_sample_and_analyze())
+
+                # Print the results for each level
+                for level, (histograms, metadata) in results:
+                    imgw, imgh, tilesw, tilesh = metadata
+                    print(f"  Level {level}:")
+                    print(f"    Image dimensions: {imgw} x {imgh}")
+                    print(f"    Tiles: {tilesw} x {tilesh}")
+                    print(f"    Histograms shape: {histograms.shape}\n\n")
+            else:
+                print(f"\nSkipping analysis file already exists:\n\t{img_file}\n")
+
+        print("\nAnalysis of all images completed successfully.")
+
+if __name__ == "__main__":
+    # demo()
+    analyze_all_images(limit=999, dryrun=False)

scripts/image_files_to_folders.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# Directory containing the JPEG images
+IMG_DIR="img"
+
+# Check if the directory exists
+if [ ! -d "$IMG_DIR" ]; then
+    echo "Directory $IMG_DIR does not exist."
+    exit 1
+fi
+
+# Loop through each JPEG file in the directory
+for img_file in "$IMG_DIR"/*.jpg "$IMG_DIR"/*.jpeg; do
+    # Check if the file exists (to handle cases where no files match the pattern)
+    if [ -f "$img_file" ]; then
+        # Extract the base name of the file (without extension)
+        base_name=$(basename "$img_file" | cut -d. -f1)
+
+        # Create a folder with the same base name
+        mkdir -p "$IMG_DIR/$base_name"
+
+        # Move the image file into the folder
+        mv "$img_file" "$IMG_DIR/$base_name/"
+
+        echo "Moved $img_file to $IMG_DIR/$base_name/"
+    fi
+done
+
+echo "Done!"

scripts/insert_histogram_data.py

@@ -0,0 +1,89 @@
+"""
+Script to iterate over histogram data and insert it into the database.
+This is a stub for future development.
+"""
+import os
+import re
+import numpy as np
+import sqlalchemy
+from sqlalchemy import text, exc
+
+# Database URL from alembic.ini
+DATABASE_URL = "postgresql+psycopg2://postgres:pgRocks@localhost:5432/testdb"
+BASE_DIR = '/home/kfl/dev/git/avdb-py-v1'
+
+def insert_histogram_data():
+    """
+    Iterate over histogram data for levels 3-5 and insert it into the database.
+    This is a stub and will be developed over several prompts.
+    """
+    engine = sqlalchemy.create_engine(DATABASE_URL)
+    
+    # Placeholder for histogram data directory
+    img_folder = "img"
+    inserted_count = 0
+    skipped_count = 0
+    duplicate_count = 0
+    
+    # Iterate over histogram data files (stub)
+    for root, dirs, files in os.walk(img_folder):
+        for file in files:
+            if file.lower().endswith('.mmap'):
+                histogram_file_path = os.path.join(BASE_DIR, root, file)
+                print(histogram_file_path)
+                level = 0
+                m = re.match(r'.*(\d{1})\.jpeg.histos.mmap', histogram_file_path)
+                if m:
+                    level = int(m.group(1))
+                if level in (3,4,5):
+                    # Extract histogram metadata (stub)
+                    try:
+                        histogram_values = []  # Placeholder
+
+                        gridsplit = 0
+                        if level == 3:
+                            gridsplit = 8
+                        elif level == 4:
+                            gridsplit = 4
+                        elif level == 5:
+                            gridsplit = 2
+
+                        data = np.memmap(
+                            histogram_file_path,
+                            dtype="float32",
+                            mode='r+',
+                            shape=((gridsplit+1)**2,48)
+                        )
+                        print(data.shape)
+                        for row in data:
+                            print(row)
+                    
+                        # # Insert into the database (stub)
+                        # with engine.connect() as connection:
+                        #     query = text(
+                        #         """
+                        #         INSERT INTO histogram_table (level, histogram_values)
+                        #         VALUES (:level, :histogram_values)
+                        #         """
+                        #     )
+                        #     connection.execute(
+                        #         query,
+                        #         {"level": level, "histogram_values": histogram_values}
+                        #     )
+                        #     connection.commit()
+                        #     inserted_count += 1
+                        #     print(f"Inserted histogram data for level {level}")
+                    except exc.IntegrityError as e:
+                        # Handle duplicate key violations
+                        print(f"Duplicate histogram entry skipped for {histogram_file_path}: {e}")
+                        duplicate_count += 1
+                    except Exception as e:
+                        print(f"Failed to insert histogram data for {histogram_file_path}: {e}")
+                        skipped_count += 1
+        
+    print(f"\nTotal histogram entries inserted: {inserted_count}")
+    print(f"Total duplicates skipped: {duplicate_count}")
+    print(f"Total entries skipped due to errors: {skipped_count}")
+
+if __name__ == "__main__":
+    insert_histogram_data()

scripts/insert_images_into_db.py

@@ -0,0 +1,74 @@
+"""
+Script to iterate over all images in the img folder and insert their metadata into the avdb_image table.
+"""
+import os
+from PIL import Image
+import sqlalchemy
+from sqlalchemy import text, exc
+
+# Database URL from alembic.ini
+DATABASE_URL = "postgresql+psycopg2://postgres:pgRocks@localhost:5432/testdb"
+
+def get_image_metadata(image_path):
+    """Extract metadata (width, height) from an image."""
+    with Image.open(image_path) as img:
+        width, height = img.size
+    return width, height
+
+def insert_images_into_db():
+    """Iterate over all images in the img folder and insert their metadata into the avdb_image table."""
+    engine = sqlalchemy.create_engine(DATABASE_URL)
+
+    # Iterate over all subdirectories in the img folder
+    img_folder = "img"
+    inserted_count = 0
+    skipped_count = 0
+    test_shim = 'TESTIMG.jpeg'
+    duplicate_count = 0
+    
+    for root, dirs, files in os.walk(img_folder):
+        for file in files:
+            # test nmode
+            if file.endswith(test_shim):
+            # if file.lower().endswith(('.jpg', '.jpeg')):
+                image_path = os.path.join(root, file)
+                img_name = file
+                
+                # Extract width and height
+                try:
+                    width, height = get_image_metadata(image_path)
+                except Exception as e:
+                    print(f"Failed to extract metadata for {image_path}: {e}")
+                    skipped_count += 1
+                    continue
+                
+                # Insert into the database
+                try:
+                    with engine.connect() as connection:
+                        query = text(
+                            """
+                            INSERT INTO avdb_image (img_path, img_name, width, height)
+                            VALUES (:img_path, :img_name, :width, :height)
+                            """
+                        )
+                        connection.execute(
+                            query,
+                            {"img_path": image_path, "img_name": img_name, "width": width, "height": height}
+                        )
+                        connection.commit()
+                        inserted_count += 1
+                        print(f"Inserted: {img_name} (Width: {width}, Height: {height})")
+                except exc.IntegrityError:
+                    # Handle duplicate key violations
+                    print(f"Duplicate entry skipped for {image_path}")
+                    duplicate_count += 1
+                except Exception as e:
+                    print(f"Failed to insert {image_path} into the database: {e}")
+                    skipped_count += 1
+    
+    print(f"\nTotal images inserted: {inserted_count}")
+    print(f"Total duplicates skipped: {duplicate_count}")
+    print(f"Total images skipped due to errors: {skipped_count}")
+
+if __name__ == "__main__":
+    insert_images_into_db()

scripts/pyramiddata_2tiles_insert.sql

@@ -0,0 +1,45 @@
+insert into pyramiddata_2tiles (
+pyrdata_id, img_id, level, tilex_a, tiley_a, x_a, y_a, x0_a, y0_a, w_a, h_a,
+tilex_b, tiley_b, x_b, y_b, x0_b, y0_b, w_b, h_b, hash, tile_data
+)
+select a.pyrdata_id as pyrdata_id
+	,a.img_id as img_id
+	,a.level as level 
+	,a.tilex as tilex_a
+	,a.tiley as tiley_a
+	,a.x as x_a
+	,a.y as y_a
+	,a.x0 as x0_a
+	,a.y0 as y0_a
+	,a.w as w_a
+	,a.h as h_a
+	,b.tilex as tilex_b
+	,b.tiley as tilex_b
+	,a.x as x_b
+	,a.y as y_b
+	,b.x0 as x0_b
+	,b.y0 as y0_b
+	,b.w as w_b
+	,b.h as h_b
+	,a.hash as hash
+	,a.tile_data||b.tile_data as tile_data
+from (
+select *
+from pyramiddata
+where level >= 3
+and level <= 6
+--and img_id = 1
+and tile_data is not null
+) a
+JOIN (
+select *
+from pyramiddata
+where level >= 4
+and level <= 7
+--and img_id = 1
+and tile_data is not null
+) b
+on a.img_id = b.img_id
+and a.level + 1 = b.level
+and a.tilex / 2 = b.tilex
+and a.tiley / 2 = b.tiley

scripts/rm-all-analysis-files.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Navigate to the img directory
+cd img || exit
+
+# Loop through each folder in the img directory
+for folder in IMG_*; do
+    # Check if the folder exists and is a directory
+    if [ -d "$folder" ]; then
+        # Navigate into the folder
+        cd "$folder" || continue
+
+        for file in *; do
+            if [[ -f "$file" ]] && ! [[ "$file" =~ ^IMG_[0-9]{4}\.jpeg$ ]]; then
+                rm "$file"
+            fi
+        done
+
+        # Navigate back to the img directory
+        cd ..
+    fi
+done

scripts/scratch.py

@@ -0,0 +1,33 @@
+# def iter_and_append_from_csv(csvfile):
+#     """
+#     Read and iterate over a CSV file.
+#     Args:
+#         csvfile (file): A file object representing the CSV file to read.
+    
+#     Yields:
+#         list: A row from the CSV file as a list of strings.
+#     """
+#     csvreader = csv.reader(csvfile)
+#     for row in csvreader:
+#         yield row
+
+# def _read_csv_data(filename: str, row_indices: List[int], col_indices: List[int]) -> List[List[str]]:
+#     """
+#     Helper function to read and extract specific rows and columns from a CSV file.
+    
+#     Args:
+#         filename (str): Path to the CSV file.
+#         row_indices (List[int]): List of row indices to extract.
+#         col_indices (List[int]): List of column indices to extract.
+    
+#     Returns:
+#         List[List[str]]: A list of rows, where each row is a list of values at the specified column indices.
+#     """
+#     try:
+#         with open(filename, 'r', encoding='utf-8') as csvfile:
+#             csvfile_data = list(iter_and_append_from_csv(csvfile))
+#             return [[csvfile_data[row][col] for col in col_indices] for row in row_indices]
+#     except FileNotFoundError as e:
+#         print(e)
+#         return []
+

scripts/test_proccess_one_image.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Script to test the process_image_analysis function in db.py
+# This script will run the function and stop on errors.
+
+# Define the path to the Python script and the image to process
+# PYTHON_SCRIPT="src/db.py"
+# Replace with the actual image path
+IMAGE_PATH="/home/kfl/dev/git/avdb-py-v1/img/TESTIMG/TESTIMG.jpeg"
+
+# Function to run the test
+test_process_image_analysis() {
+    echo "Starting test for process_image_analysis..."
+
+    # Run the Python script and capture the output
+    output=$(python3 -c "
+import sys
+sys.path.append('src')
+from db import DBOps, DBConnection
+
+# Initialize the database connection
+db_conn = DBConnection('testdb')
+db_ops = DBOps(db_conn)
+
+# Run the process_image_analysis function
+try:
+    # raise(ValueError('testerr'))
+    db_ops.process_image_analysis('$IMAGE_PATH')
+    print('process_image_analysis completed successfully.')
+except Exception as e:
+    print(f'Error in process_image_analysis: {e}')
+    sys.exit(1)
+" 2>&1 | tee error_log.txt
+)
+    # Check the output for errors
+    if [[ "$output" == *"Error"* ]]; then
+        echo "Test failed: $output"
+        exit 1
+    else
+        echo "Test passed: $output"
+    fi
+}
+
+# Run the test
+test_process_image_analysis
+
+# Exit with the appropriate status
+# exit $?

tests/TEST.md

@@ -0,0 +1,51 @@
+# Test Plan for `ColorFeaturesPyramid`
+
+## Overview
+This document outlines the plan for testing and documenting the `ColorFeaturesPyramid` class.
+
+## 1. Understanding the `ColorFeaturesPyramid` Class
+- The `ColorFeaturesPyramid` class performs pyramid down-sampling and color feature analysis on images.
+- It relies on the `ColorFeaturesTile` class for tile-based analysis.
+- Key parameters:
+  - `uuid`: Identifier for the image.
+  - `tilew` and `tileh`: Dimensions of the tiles for analysis.
+  - `down_sampling`: Number of down-sampling levels.
+  - `colorspace`: Color space for analysis (e.g., `lab`, `rgb`, `hsv`).
+- The class reads an image, down-samples it, and analyzes the tiles at each level.
+
+## 2. Design the Test Function
+### Test Scope
+Unit tests to verify the functionality of the `ColorFeaturesPyramid` class.
+
+### Test Cases
+1. **Initialization Test**:
+   - Verify that the class initializes correctly with default and custom parameters.
+   - Ensure that the `check_params` function from `utils.py` is used correctly.
+2. **Down-Sampling Test**:
+   - Verify that the down-sampling process works as expected.
+   - Check that the down-sampled images are saved correctly.
+3. **Tile Analysis Test**:
+   - Verify that the tile analysis produces the expected output.
+   - Ensure that the histograms are computed correctly.
+4. **Edge Cases**:
+   - Test with invalid inputs (e.g., non-existent image, invalid color space).
+   - Test with extreme values for parameters (e.g., very large or small tile sizes).
+
+### Testing Framework
+Use `pytest` for writing and running the tests.
+
+## 3. Document the Test Function
+- Write clear docstrings for the test function.
+- Include examples of how to use the `ColorFeaturesPyramid` class.
+- Document the expected behavior and assumptions.
+- Add comments to explain the purpose of each test case.
+
+## 4. Plan the Implementation
+- Create a new test file: `test_color_features_pyramid.py`.
+- Implement the test function and test cases.
+- Ensure the test function is modular and reusable.
+- Use the `check_params` function from `utils.py` to validate parameters.
+
+## 5. Verify the Plan
+- Review the plan to ensure all aspects are covered.
+- Confirm that the test function aligns with the purpose of the `ColorFeaturesPyramid` class.

tests/__init__.py

@@ -0,0 +1,3 @@
+# tests/__init__.py
+
+# This file can be empty or include test-specific initialization

tests/run_tests.py

@@ -0,0 +1,8 @@
+import unittest
+
+if __name__ == '__main__':
+    # Discover and run all tests in the tests directory
+    loader = unittest.TestLoader()
+    suite = loader.discover('tests', pattern='test_*.py')
+    runner = unittest.TextTestRunner()
+    runner.run(suite)

tests/test_alembic_migration.py

@@ -0,0 +1,52 @@
+"""
+Test script to verify the Alembic migration for the avdb_image table.
+"""
+import sqlalchemy
+from sqlalchemy import inspect, text
+
+# Database URL from alembic.ini
+DATABASE_URL = "postgresql+psycopg2://postgres:pgRocks@localhost:5432/testdb"
+
+def test_avdb_image_table_exists():
+    """Test if the avdb_image table exists in the database."""
+    engine = sqlalchemy.create_engine(DATABASE_URL)
+    inspector = inspect(engine)
+    
+    # Print database and table configuration
+    print("Database URL:", DATABASE_URL)
+    print("\nChecking for avdb_image table...")
+    
+    # Check if the table exists
+    assert inspector.has_table("avdb_image"), "avdb_image table does not exist!"
+    print("✓ avdb_image table exists.")
+    
+    # Verify the table structure
+    columns = inspector.get_columns("avdb_image")
+    column_names = [column["name"] for column in columns]
+    
+    print("\nColumns in avdb_image table:")
+    for column in columns:
+        print(f"  - {column['name']}: {column['type']}, Nullable: {column['nullable']}")
+    
+    expected_columns = ["img_id", "img_path", "img_name", "width", "height", "created_at"]
+    for col in expected_columns:
+        assert col in column_names, f"Column '{col}' not found in avdb_image table!"
+    print("\n✓ All expected columns are present in the avdb_image table.")
+    
+    # Verify primary key
+    primary_key = inspector.get_pk_constraint("avdb_image")
+    assert primary_key is not None, "Primary key not found in avdb_image table!"
+    assert primary_key["constrained_columns"] == ["img_id"], "Primary key is not set on img_id!"
+    print("✓ Primary key is correctly set on img_id.")
+    
+    # Verify nullable constraints
+    nullable_columns = [col["name"] for col in columns if col["nullable"]]
+    non_nullable_columns = ["img_path", "img_name", "width", "height"]
+    for col in non_nullable_columns:
+        assert col not in nullable_columns, f"Column '{col}' should not be nullable!"
+    print("✓ Non-nullable constraints are correctly set.")
+    
+    print("\nAll tests passed! The avdb_image table was created successfully.")
+
+if __name__ == "__main__":
+    test_avdb_image_table_exists()

tests/test_color_features_pyramid.py

@@ -0,0 +1,150 @@
+"""
+Test module for the ColorFeaturesPyramid class.
+
+This module contains unit tests to verify the functionality of the ColorFeaturesPyramid class,
+including initialization, down-sampling, and tile analysis.
+"""
+
+import os
+import sys
+import pytest
+import cv2 as cv
+import numpy as np
+
+from color_features_pyramid import ColorFeaturesPyramid
+
+
+class TestColorFeaturesPyramid:
+    """
+    Test suite for the ColorFeaturesPyramid class.
+    """
+
+    @pytest.fixture
+    def setup_image(self, tmp_path):
+        """
+        Fixture to set up a test image for the ColorFeaturesPyramid tests.
+        
+        Args:
+            tmp_path: Pytest fixture providing a temporary directory.
+        
+        Returns:
+            tuple: Path to the test image and the image UUID.
+        """
+        # Create a temporary directory for the test image
+        img_dir = "/home/kfl/dev/git/avdb-py-v1/img"
+        
+        # Create a simple test image
+        img = np.zeros((100, 100, 3), dtype=np.uint16)
+        img[:, :] = (255, 0, 0)  # Red image
+        
+        img_path = img_dir + "/TESTIMG.jpeg"
+        # cv.imwrite(str(img_path), img)
+        
+        return str(img_path), "TESTIMG"
+
+    def test_initialization_default_params(self, setup_image):
+        """
+        Test that the ColorFeaturesPyramid class initializes correctly with default parameters.
+        
+        Args:
+            setup_image: Fixture providing the test image path and UUID.
+        """
+        img_path, uuid = setup_image
+        
+        # Initialize with default parameters
+        pyramid = ColorFeaturesPyramid(uuid=uuid, root_path=os.path.dirname(img_path))
+        
+        # Verify default parameters
+        assert pyramid.uuid == uuid
+        assert pyramid.down_sampling_levels == 9
+        assert pyramid.colorspace == hsv
+
+    def test_initialization_custom_params(self, setup_image):
+        """
+        Test that the ColorFeaturesPyramid class initializes correctly with custom parameters.
+        
+        Args:
+            setup_image: Fixture providing the test image path and UUID.
+        """
+        img_path, uuid = setup_image
+        
+        # Initialize with custom parameters
+        pyramid = ColorFeaturesPyramid(
+            uuid=uuid,
+            root_path=os.path.dirname(img_path),
+            down_sampling=5,
+            colorspace="rgb"
+        )
+        
+        # Verify custom parameters
+        assert pyramid.uuid == uuid
+        assert pyramid.down_sampling_levels == 5
+        assert pyramid.colorspace == "rgb"
+
+    def test_down_sampling(self, setup_image):
+        """
+        Test that the down-sampling process works as expected.
+        
+        Args:
+            setup_image: Fixture providing the test image path and UUID.
+        """
+        img_path, uuid = setup_image
+
+        # Initialize the pyramid
+        pyramid = ColorFeaturesPyramid(uuid=uuid, root_path=os.path.dirname(img_path))
+        
+        # Run the down-sampling and analysis process
+        results = list(pyramid._down_sample_and_analyze())
+        
+        # Verify that results are generated for each level
+        assert len(results) > 0
+        
+        # Verify that down-sampled images are saved
+        for level in range(1, pyramid.down_sampling_levels):
+            # Construct the correct path for the down-sampled image
+            ds_img_filename = f"{uuid}/{uuid}.{pyramid.colorspace}.pyr_{level}.jpeg"
+            ds_img_path = os.path.join(os.path.dirname(img_path), ds_img_filename)
+            assert os.path.exists(ds_img_path), f"Down-sampled image not found: {ds_img_path}"
+
+    def test_tile_analysis(self, setup_image):
+        """
+        Test that the tile analysis produces the expected output.
+        
+        Args:
+            setup_image: Fixture providing the test image path and UUID.
+        """
+        img_path, uuid = setup_image
+        
+        # Initialize the pyramid
+        pyramid = ColorFeaturesPyramid(uuid=uuid, root_path=os.path.dirname(img_path))
+        
+        # Run the down-sampling and analysis process
+        results = list(pyramid._down_sample_and_analyze())
+        
+        # Verify that results contain histograms and metadata
+        for level, (histograms, metadata) in results:
+            assert histograms is not None
+            assert len(metadata) == 4  # (imgw, imgh, tilesw, tilesh)
+
+    def test_edge_case_invalid_image(self):
+        """
+        Test that the class handles invalid image paths gracefully.
+        """
+        # Initialize with a non-existent image
+        with pytest.raises(Exception):
+            pyramid = ColorFeaturesPyramid(uuid="nonexistent_image", root_path="/tmp")
+            list(pyramid._down_sample_and_analyze())
+
+    def test_edge_case_invalid_colorspace(self, setup_image):
+        """
+        Test that the class handles invalid color spaces gracefully.
+        
+        Args:
+            setup_image: Fixture providing the test image path and UUID.
+        """
+        img_path, uuid = setup_image
+        
+        # Initialize with an invalid color space
+        with pytest.raises(Exception):
+            pyramid = ColorFeaturesPyramid(uuid=uuid, root_path=os.path.dirname(img_path), colorspace="invalid")
+            list(pyramid._down_sample_and_analyze())

tests/test_db.py

@@ -0,0 +1,131 @@
+import unittest
+import os
+import numpy as np
+import csv
+import uuid
+from unittest.mock import MagicMock, patch
+import psycopg2
+from psycopg2 import extras
+
+from avdb.db import DBOps
+from avdb.analysis import convert_mmap_to_csv
+
+class TestDBOps(unittest.TestCase):
+    def setUp(self):
+        # Mock the database connection and cursor
+        self.mock_conn = MagicMock()
+        self.mock_cursor = MagicMock()
+        self.mock_conn.get_cursor.return_value = self.mock_cursor
+        self.mock_conn.get_connection.return_value = self.mock_conn
+
+        # Initialize DBOps with the mocked connection
+        self.db_ops = DBOps(self.mock_conn)
+
+    def tearDown(self):
+        # Clean up any temporary files created during tests
+        if os.path.exists("test_histogram.mmap"):
+            os.remove("test_histogram.mmap")
+        if os.path.exists("test_histogram.csv"):
+            os.remove("test_histogram.csv")
+        if os.path.exists("test_tile.mmap"):
+            os.remove("test_tile.mmap")
+
+    def test_convert_mmap_to_csv(self):
+        # Create a temporary .mmap file for testing
+        mmap_file_path = "test_histogram.mmap"
+        csv_file_path = "test_histogram.csv"
+
+        # Create dummy data for the .mmap file
+        dummy_data = np.random.rand(81, 48).astype('float32')
+        with open(mmap_file_path, 'wb') as f:
+            f.write(dummy_data.tobytes())
+
+        # Call the conversion function
+        convert_mmap_to_csv(mmap_file_path, csv_file_path)
+
+        # Verify the CSV file was created and contains the correct data
+        with open(csv_file_path, 'r') as csv_file:
+            reader = csv.reader(csv_file)
+            csv_data = list(reader)
+
+        # Convert the CSV data back to a numpy array for comparison
+        csv_data_array = np.array(csv_data, dtype='float32')
+
+        # Assert that the data matches
+        np.testing.assert_array_equal(csv_data_array, dummy_data)
+
+    def test_insert_tile_data(self):
+        # Create dummy tile data
+        dummy_rows = [
+            (
+                1,  # img_id
+                3,  # level
+                0,  # tilex
+                0,  # tiley
+                100,  # x
+                100,  # y
+                50,  # x0
+                50,  # y0
+                10,  # w
+                10,  # h
+                str(uuid.uuid4()),  # hash
+                [1, 2, 3]  # tile_data
+            ),
+            (
+                1,  # img_id
+                3,  # level
+                1,  # tilex
+                1,  # tiley
+                200,  # x
+                200,  # y
+                100,  # x0
+                100,  # y0
+                10,  # w
+                10,  # h
+                str(uuid.uuid4()),  # hash
+                [4, 5, 6]  # tile_data
+            )
+        ]
+
+        # Mock the execute_batch method
+        with patch('avdb.db.psycopg2.extras.execute_batch') as mock_execute_batch:
+            # Call the insert_tile_data method
+            self.db_ops.insert_tile_data(dummy_rows)
+
+            # Verify that execute_batch was called with the correct query and data
+            mock_execute_batch.assert_called_once_with(
+                self.mock_cursor,
+                """
+                INSERT INTO pyramiddata_level3 (
+                    img_id, level, tilex, tiley, x, y, x0, y0, w, h, hash, tile_data
+                )
+                VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                """,
+                dummy_rows
+            )
+        
+    def test_process_and_insert_tile_data_batched(self):
+        # Create a temporary .mmap file for testing
+        mmap_file_path = "test_tile.mmap"
+        dummy_data = np.random.randint(0, 255, size=(100, 3), dtype=np.uint16)
+        with open(mmap_file_path, 'wb') as f:
+            f.write(dummy_data.tobytes())
+
+        # Mock the load_memmapped_tile_data method to return the dummy data
+        self.db_ops.load_memmapped_tile_data = MagicMock(return_value=dummy_data)
+
+        # Mock the execute_batch method
+        with patch('avdb.db.psycopg2.extras.execute_batch') as mock_execute_batch:
+            # Call the process_and_insert_tile_data_batched method
+            self.db_ops.process_and_insert_tile_data_batched(
+                img_id=1,
+                img_dims=(1000, 1000),
+                level=3,
+                tile_file=mmap_file_path
+            )
+
+            # Verify that execute_batch was called
+            mock_execute_batch.assert_called()
+
+if __name__ == '__main__':
+    unittest.main()