Read data from azure data lake using python


Asked by wiki @ & viewed by 9 persons


Python code to access Azure Data Lake Store

I 'm looking at Microsoft Documentation here and here, I have created Web App in Azure Active Directory to access the Data Lake Store

From the Web App I have Object ID, Application ID and Key

looking at the documentations I see this:

adlCreds = lib.auth(tenant_id = 'FILL-IN-HERE', client_secret = 'FILL-IN-HERE', client_id = 'FILL-IN-HERE', resource = 'https://datalake.azure.net/')

how to use it to authenticate my code and run operation on Data Lake Store?

here is my full test code:

## Use this for Azure AD authentication
from msrestazure.azure_active_directory import AADTokenCredentials

## Required for Azure Data Lake Store account management
from azure.mgmt.datalake.store import DataLakeStoreAccountManagementClient
from azure.mgmt.datalake.store.models import DataLakeStoreAccount

## Required for Azure Data Lake Store filesystem management
from azure.datalake.store import core, lib, multithread

# Common Azure imports
import adal
from azure.mgmt.resource.resources import ResourceManagementClient
from azure.mgmt.resource.resources.models import ResourceGroup

## Use these as needed for your application
import logging, getpass, pprint, uuid, time


## Declare variables
subscriptionId = 'FILL-IN-HERE'
adlsAccountName = 'FILL-IN-HERE'

tenant_id = 'FILL-IN-HERE'
client_secret = 'FILL-IN-HERE'
client_id = 'FILL-IN-HERE'


## adlCreds = lib.auth(tenant_id = 'FILL-IN-HERE', client_secret = 'FILL-IN-HERE', client_id = 'FILL-IN-HERE', resource = 'https://datalake.azure.net/')
from azure.common.credentials import ServicePrincipalCredentials
adlCreds = lib.auth(tenant_id, client_secret, client_id, resource = 'https://datalake.azure.net/')


## Create a filesystem client object
adlsFileSystemClient = core.AzureDLFileSystem(adlCreds, store_name=adlsAccountName)

## Create a directory
adlsFileSystemClient.mkdir('/mysampledirectory')

when I try to ru the code I get error:

[Running] python "c:....\dls.py" Traceback (most recent call last): File "c:....\dls.py", line 38, in adlCreds = lib.auth(tenant_id, client_secret, client_id, resource = 'https://datalake.azure.net/') File "C:\Python36\lib\site-packages\azure\datalake\store\lib.py", line 130, in auth password, client_id) File "C:\Python36\lib\site-packages\adal\authentication_context.py", line 145, in acquire_token_with_username_password return self._acquire_token(token_func) File "C:\Python36\lib\site-packages\adal\authentication_context.py", line 109, in _acquire_token return token_func(self) File "C:\Python36\lib\site-packages\adal\authentication_context.py", line 143, in token_func return token_request.get_token_with_username_password(username, password) File "C:\Python36\lib\site-packages\adal\token_request.py", line 280, in get_token_with_username_password self._user_realm.discover() File "C:\Python36\lib\site-packages\adal\user_realm.py", line 152, in discover raise AdalError(return_error_string, error_response) adal.adal_error.AdalError: User Realm Discovery request returned http error: 404 and server response:

404 - File or directory not found.

Server Error

404 - File or directory not found.

The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.

[Done] exited with code=1 in 1.216 seconds

Do you know the better answer?

Similar questions

Python read data from arduino

Asked by wiki @ & viewed by 5 persons

python to arduino serial read & write I'm trying to "ping pong" info back and forth between some python code …

Read data into array python

Asked by wiki @ & viewed by 8 persons

Python reading data file into lists or arrays For some reason, I decided to save my list of data as …

Read data from website python

Asked by wiki @ & viewed by 8 persons

extract data from website using python I recently started learning python and one of the first projects I did was …

How to read email from outlook using python

Asked by wiki @ & viewed by 8 persons

Reading e-mails from Outlook with Python through MAPI I'm trying to write a short program that will read in the …

Azure data studio python

Asked by wiki @ & viewed by 8 persons

Unable to open pandas python package from Azure Data Studio, while configuring SQL Server 2019 Big Data Cluster I'm working …

How to read excel file in python using jupyter notebook

Asked by wiki @ & viewed by 11 persons

Reading xlsx file using jupyter notebook I have tried to read an xlsx file which is stored in my system …

Use python to read excel file

Asked by wiki @ & viewed by 21 persons

Reading Excel File using Python, how do I get the values of a specific column with indicated column name? I've …

Python read data from url

Asked by wiki @ & viewed by 13 persons

How can I read the contents of an URL with Python? The following works when I paste it on the …

How to read data from a webpage in python

Asked by wiki @ & viewed by 14 persons

How can I read the contents of an URL with Python? The following works when I paste it on the …

Read data from url in python

Asked by wiki @ & viewed by 25 persons

How can I read the contents of an URL with Python? The following works when I paste it on the …

Python read csv data type

Asked by wiki @ & viewed by 29 persons

Read data from CSV file and transform from string to correct data-type, including a list-of-integer column When I read data …

How to read mail using python

Asked by wiki @ & viewed by 14 persons

How to read email using Python 3 Now I am here import imaplib mail = imaplib.IMAP4_SSL('imap.gmail.com') mail.login('login@gmail.com', 'password') mail.list() # …

Read excel in python using pandas

Asked by wiki @ & viewed by 42 persons

Reading an Excel file in python using pandas I am trying to read an excel file this way : newFile …

How to read excel file in python using xlsxwriter

Asked by wiki @ & viewed by 41 persons

Is it possible to read data from an Excel sheet in Python using Xlsxwriter? If so how? I'm doing the …

How to read excel file in python using pandas

Asked by wiki @ & viewed by 53 persons

Reading an Excel file in python using pandas I am trying to read an excel file this way : newFile …

Most viewed questions


Python strftime milliseconds

Asked by wiki @ & viewed by 201 persons

Format a datetime into a string with milliseconds I want to have a datetime string from the date with milliseconds. …

What does the percent sign mean in python

Asked by wiki @ & viewed by 136 persons

What does the percentage sign mean in Python In the tutorial there is an example for finding prime numbers: >>> …

Self keyword in python

Asked by wiki @ & viewed by 135 persons

What is the purpose of the word 'self'? What is the purpose of the self word in Python? I understand …

Check object type python

Asked by wiki @ & viewed by 133 persons

Determine the type of an object? Is there a simple way to determine if a variable is a list, dictionary, …

Correlation matrix python

Asked by wiki @ & viewed by 132 persons

Plot correlation matrix using pandas I have a data set with huge number of features, so analysing the correlation matrix …

Alphabetical sort python

Asked by wiki @ & viewed by 131 persons

Python data structure sort list alphabetically I am a bit confused regarding data structure in python; (),[], and {}. I …

Python random

Asked by wiki @ & viewed by 127 persons

Generate random integers between 0 and 9 How can I generate random integers between 0 and 9 (inclusive) in Python? …

Python one line for loop

Asked by wiki @ & viewed by 124 persons

Python one-line “for” expression I'm not sure if I need a lambda, or something else. But still, I need the …

Python create dataframe from list

Asked by wiki @ & viewed by 124 persons

Python: create a pandas data frame from a list I am using the following code to create a data frame …

How to add a column to dataframe python

Asked by wiki @ & viewed by 123 persons

Adding new column to existing DataFrame in Python pandas I have the following indexed DataFrame with named columns and rows …

Why we use self in python

Asked by wiki @ & viewed by 116 persons

What is the purpose of the word 'self'? What is the purpose of the self word in Python? I understand …

Shape 0 python

Asked by wiki @ & viewed by 115 persons

x.shape[0] vs x[0].shape in NumPy Let say, I have an array with x.shape = (10,1024) when I try to print …

Python print working directory

Asked by wiki @ & viewed by 113 persons

Find current directory and file's directory [duplicate] This question already has answers here: How do you properly determine the current …

Image shape python

Asked by wiki @ & viewed by 110 persons

Python OpenCV2 (cv2) wrapper to get image size? How to get the size of an image in cv2 wrapper in …

Interface grafica python

Asked by wiki @ & viewed by 108 persons

Como criar interface gráfica em Python? [duplicada] Essa pergunta já tem uma resposta aqui: É possível criar interfaces gráficas para …