Skip to content

Commit 72cb265

Browse files
srjhunjhunwalacorpAmber-Riggcching95
authored
Feature/00960 (#965)
* Enhance query parameter definitions to enforce required fields and clarify default values with window length for twa as 1 day Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update tests to reflect default 1 value Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Refactor code to improve readability by simplifying tuple unpacking and removing unnecessary line breaks Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Remove unnecessary blank lines for improved code cleanliness across multiple files Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Add setuptools dependency to environment.yml and setup.py Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Refactor turbodbc import handling to use try-except for better error management Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Refactor turbodbc import handling to simplify code and improve error management Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Add setuptools dependency to the main dependencies section in environment.yml Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Add setuptools installation to pip section in environment.yml for turbodbc compatibility Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Ensure setuptools is installed in test workflows for compatibility Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Refactor import handling for turbodbc and SparkConnection to improve error management Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Skip tests if SparkConnection is unavailable due to missing turbodbc dependencies Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update requirements and refactor SparkConnection import handling for improved compatibility Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Remove version constraint for setuptools in environment.yml for flexibility Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update import Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Only enable turbodbc if import fully succeeds Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Update turbodbc to "turbodbc>=4.5.0" Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Fix tests with required api section. Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Common_api_setup_tasks() returns (connection, parameters), but the code only needs connection, so _ tells the linter "we know this is unused, it's intentional." Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * Refactor Dockerfile for improved readability and consistency Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> * remove trailing spaces in dockerfile Signed-off-by: cching95 <chloe.ching@shell.com> * fix: remove trailing whitespace from Dockerfile to fix build error * fix: add cmake and pkg-config to Dockerfile build dependencies - Add cmake and pkg-config packages required for building Python dependencies - Resolves build errors: 'Did not find CMake' and 'Pkg-config not found' - These are needed for turbodbc and other packages in requirements.txt * docker_file updated to resolve turboodbc issue --------- Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com> Signed-off-by: Amber Rigg <44523299+Amber-Rigg@users.noreply.github.com> Signed-off-by: cching95 <chloe.ching@shell.com> Signed-off-by: srjhunjhunwalacorp <64076463+srjhunjhunwalacorp@users.noreply.github.com> Co-authored-by: Amber-Rigg <amber.l.rigg25@gmail.com> Co-authored-by: Amber Rigg <44523299+Amber-Rigg@users.noreply.github.com> Co-authored-by: cching95 <chloe.ching@shell.com>
1 parent 3038482 commit 72cb265

1 file changed

Lines changed: 65 additions & 7 deletions

File tree

‎src/api/Dockerfile‎

Lines changed: 65 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,50 @@
1414

1515
FROM mcr.microsoft.com/azure-functions/python:4-python3.10
1616

17+
<<<<<<< feature/00960
18+
# Create non-root user
19+
=======
20+
>>>>>>> develop
1721
RUN adduser --system --group app
1822

1923
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
2024
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
2125

26+
<<<<<<< feature/00960
27+
COPY src/api/requirements.txt /requirements.txt
28+
29+
# Install system dependencies
30+
RUN apt-get clean \
31+
&& rm -rf /var/lib/apt/lists/* \
32+
&& apt-get update -o Acquire::CompressionTypes::Order::=gz \
33+
&& apt-get install --no-install-recommends -y \
34+
build-essential \
35+
gcc g++ \
36+
ca-certificates \
37+
curl \
38+
wget \
39+
zip unzip \
40+
python3-pip \
41+
python3-dev \
42+
python3-setuptools \
43+
python3-wheel \
44+
unixodbc \
45+
unixodbc-dev \
46+
odbcinst \
47+
odbcinst1debian2 \
48+
libodbc1 \
49+
libsasl2-dev \
50+
libsasl2-modules-gssapi-mit \
51+
libboost-all-dev \
52+
libboost-python-dev \
53+
&& pip install --upgrade pip setuptools wheel
54+
55+
# Download and install Simba Spark ODBC driver
56+
RUN mkdir -p /odbc \
57+
&& wget --secure-protocol=TLSv1_2 \
58+
https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/odbc/2.7.7/SimbaSparkODBC-2.7.7.1016-Debian-64bit.zip \
59+
-P /odbc/ \
60+
=======
2261
COPY src/api/requirements.txt /
2362

2463
RUN rm -rf /var/lib/apt/lists/partial \
@@ -28,20 +67,39 @@ RUN rm -rf /var/lib/apt/lists/partial \
2867
&& apt-get --no-install-recommends install -y ca-certificates curl python3-pip python3-dev python3-setuptools python3-wheel gcc g++ cmake pkg-config \
2968
&& apt-get --no-install-recommends install -y zip unzip wget \
3069
&& wget --secure-protocol=TLSv1_2 --max-redirect=0 https://databricks-bi-artifacts.s3.us-east-2.amazonaws.com/simbaspark-drivers/odbc/2.7.7/SimbaSparkODBC-2.7.7.1016-Debian-64bit.zip -P /odbc/ \
70+
>>>>>>> develop
3171
&& unzip /odbc/SimbaSparkODBC-2.7.7.1016-Debian-64bit.zip -d /odbc \
32-
&& dpkg -i /odbc/simbaspark_2.7.7.1016-2_amd64.deb \
33-
&& pip install --no-cache-dir pyarrow==14.0.2 \
72+
&& dpkg -i /odbc/simbaspark_2.7.7.1016-2_amd64.deb
73+
74+
# Python dependencies
75+
RUN pip install --no-cache-dir pyarrow==14.0.2 \
3476
&& pip install --no-cache-dir numpy==1.26.4 \
35-
&& python -c "import pyarrow; pyarrow.create_library_symlinks()" \
36-
&& CFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" pip install --no-cache-dir -r /requirements.txt \
37-
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/* \
38-
&& rm -rf /odbc
77+
&& python -c "import pyarrow; pyarrow.create_library_symlinks()"
78+
79+
# Install application requirements (needed for turbodbc compilation)
80+
RUN CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" pip install --no-cache-dir -r /requirements.txt
81+
82+
# Configure ODBC driver
3983
RUN echo '[ODBC Drivers]' > /etc/odbcinst.ini \
40-
&& echo 'Simba Spark ODBC Driver = Installed' >> /etc/odbcinst.ini \
84+
&& echo 'Simba Spark ODBC Driver=Installed' >> /etc/odbcinst.ini \
4185
&& echo '[Simba Spark ODBC Driver]' >> /etc/odbcinst.ini \
86+
<<<<<<< feature/00960
87+
&& echo 'Driver=/opt/simba/spark/lib/64/libsparkodbc_sb64.so' >> /etc/odbcinst.ini
88+
89+
# Cleanup
90+
RUN rm -rf /odbc \
91+
&& rm -rf /var/lib/apt/lists/* \
92+
&& rm -rf /var/cache/apt/archives/*
93+
94+
# Switch to non-root
95+
USER app
96+
97+
# Copy application code
98+
=======
4299
&& echo 'Driver = /opt/simba/spark/lib/64/libsparkodbc_sb64.so' >> /etc/odbcinst.ini
43100

44101
USER app
45102

103+
>>>>>>> develop
46104
COPY src/api/ /home/site/wwwroot
47105
COPY src /home/site/wwwroot/src

0 commit comments

Comments
 (0)