170 questions
0
votes
1
answer
138
views
Azure Data Factory - Dynamic Derived Column
Is the following possible in Azure Data Factory? I have a column whose value is the name of another column, and it's the value of that column that I'd like to copy into a new derived column.
e.g. I ...
- 139
0
votes
1
answer
51
views
SSIS How to convert date datatype with a size of 8 in source file with format yyyymmdd convert to yyyy-mm-dd
I am trying to load data to a SQL table using SSIS package. Text file has date columns in the format yyyymmdd, the SQL table has datatype set to date (has to remain as date), but when I'm using a ...
- 13
0
votes
1
answer
36
views
no leading zero in hours h:mm:ss time format to convert to seconds in a derived column in SSIS
a source flat file has duration columns in :00:00 format as well as 0:00:00 format. Trying to convert to seconds when there's no leading zeros in the hours. Sample of the Source file when viewed in ...
- 13
1
vote
1
answer
58
views
How to Use SSIS Derived Column to Cast a Whole Number with Positive or Negative Symbol to Float?
I have source data in a flat file that is a whole number preceded by either a positive or negative symbol. For example, +0012345 or -0012345, the first value would be translated to 123.45 while the ...
- 3,199
0
votes
1
answer
418
views
Azure Data Factory - dynamically deal with trailing column delimiters
I've got a blob container of tab delimited files that I'm trying to automate to import to am azure SQL database. One table per file
Every source file has a different schema.
I've got a for each loop ...
- 107
0
votes
1
answer
935
views
ADF: Derived column with multiple (true) conditions returning string array
I have the following problem in a Data Factory data flow:
I have two pipelines that extract data: one daily on weekdays, and one monthly on the first working day of each month.
In the data flow I ...
0
votes
2
answers
643
views
What is wrong in this statement for SSIS derived column transformation?
I am performing some transformation to create a derived column using SSIS derived column transformation task
This column "Column" to be rounded 2 decimal and put space in front.
I am using ...
- 1
1
vote
3
answers
175
views
Case expressions in derived column for SSIS
How do I modify the column having 2,4,5 as values to 2=active, 4=closd, and 5=inactv in SSIS derived column? (SQL Server DB)
I'm expecting the column should show values.
'active' instead of '2'
'closd'...
- 19
1
vote
1
answer
2k
views
ADF - Change the date format from any format coming from csv to yyyy-MM-dd HH:mm:ss while loading in target sql table taking datetime
I have a situation where I am getting dates in two separate formats, MM/dd/yyyy & yyyy-dd-MM, AND there might be even more different formats as well in csv which will be obviously in string.
Below ...
0
votes
0
answers
137
views
SSIS column skew-SSIS is concatenated to next column value to the current column when it finds the text deliminator in the column value itself
I have a ssis package which gets data from sql and dumps to CSV file.
There are half a million of rows and almost 100 columns and when i dump the data to Flat file destination, some of the column ...
- 151
0
votes
0
answers
357
views
SSIS Derived column throw's error for Decimal(17,2) datatype
Using ODBC Source to read data from Hadoop and insert them in OLE DB destination. Performing conversion operations on various columns using derived column on various column. I have Column "...
- 33
0
votes
2
answers
89
views
Adding a derived conditional column using dplyr
The initial dataset I am working with is a list of tarnatula prices overtime from two shops, one based in Canada and the other in Poland.
In order to make a fair comparison between the two shops, I ...
3
votes
1
answer
901
views
SSIS Derived Column Expression to skip null/blank value and take the next digits of string value
I am very new to SSIS and I am performing a task where I have to take the flat-file text document and then derive the columns with specific digit lengths. For eg, lines in the text document are:
...
2
votes
1
answer
283
views
Derived Column to handled mulitple conditions in CASE statement?
I've got a CASE statement that works, but because I have to do it in SSIS, I am at a loss:
SELECT [BPCNUM_0], TYPE,
CASE
WHEN [TYPE]=1 THEN 'Normal'
WHEN [TYPE]=2 THEN 'Divers'
WHEN [...
- 67
1
vote
2
answers
331
views
SSIS convert string with no 0's to date format
I'm working trying to convert a string data type to a date that does not contain 0's in the string of the month and day part and need to convert it into a date format yyyy-MM-dd, anything that is a ...
1
vote
1
answer
385
views
Derived column mapping fails
I am mapping data between my oracle table to the Azure SQL DB table. I need to map the name field from the Source which is like below
Name
AX1 BD
A1 12
BC W1
In to source, I need to just map the ...
- 1,053
-1
votes
1
answer
2k
views
SSIS - Derived Column Replace NULL Values with empty string
ISNULL([1 (Q1_1) ])? "": [1 (Q1_1) ]
When I drag the column under the Columns box in the top left, that's how my column appears,
[1 (Q1_1) ].
Why is my expression not working? I want to ...
0
votes
1
answer
689
views
Getting Data Lake Metadata in Azure Data Factory's Data Flow
I want to add the timestamp of copying parquet files to my dataframe in data flow as a derived column.
In source module I can filter parquet files by last modified which makes me think that it should ...
- 1,867
0
votes
1
answer
721
views
Dynamic column masking in Azure with derived column
I am building a data flow within azure data factory and I would like to apply some GDPR masking rules within the flow.
What I would like to do is the following: In the Derived Column (or other ...
- 1
-1
votes
1
answer
66
views
Derive column after Applying Partition by
I have data as follows:
+----+--------+------------------+
| Id | Weight | is_weight_faulty |
+----+--------+------------------+
| A | 100 | 1 |
| A | 50 | 0 |
| ...
- 394
0
votes
2
answers
2k
views
How can I turn a string into a decimal in SSIS using a derived column?
I am working in SSIS to fill a staging area database. I want to transfer the following data:
-------------------------------------------------------------------
| ShipName (String) | MaxContainers (...
0
votes
1
answer
74
views
Hive (hql) for derived columns and find the total
Could you please guide
I have a scenario where credit_Date,debit_date and loan_date can have different values of date or same. Output table have below columns
Date: should combine credit_date, ...
- 63
0
votes
1
answer
153
views
hive query for derived columns
Could you please guide.
Input table (Table1) have columns like credit_date, credit_amount, debit_date, debit_amount, loan_date, loan_amount.
Output table(Table 2) have columns like date, ...
- 63
1
vote
1
answer
1k
views
How to use derived columns in same hive table?
Could you please help me below query.
Suppose there is table employee and columns A , B and Date column.
I have to load data from table employee to another table emp with below transformation applied
...
- 63
2
votes
2
answers
6k
views
SSIS DT_STR to DT_DBDATE using Derived column
I have a fixed format .txt file which has a date field. This field may have a date or be blank. I have wracked my brains to get this field to convert from DT_STR to DT_DBDATE in the Derived Column ...
- 21